Continuous Integration of The FreeBSD Project
Hsu, Li-Wen
FreeBSD’s continuous integration project started in the late 2013. We use Jenkins automation server to build our continuous integration system. It monitors the svn repository for new commits and triggers a new build of it. In each build, the build machine compiles the latest code, creates disk image and creates a virtual machine to run test suite. In the meantime, we collect the compiler warnings and perform some further checks like clang analyzer. All these information are published to the developers and users to improve the quality of the FreeBSD project. In this talk, we describe the details of the system implementation. Continuous integration is an important and common practice of modern software engineering. The benefits of CI are developers can know the conflicts and bugs as earlier as possible, this reduces the resource consumed for resolving them later. Also, the status of the project is clearer and it is easier to estimate project progress. There are many software designed for various CI needs, among them, the most popular one is Jenkins, an open source automation server written in Java. We use Jenkins as the main component, along with some other daemons like Pure-FTPd, to build FreeBSD project’s CI system. We developed a set scripts to setup a clean and secure build environment with jails for each build. We also provide job templates for making developers to submit their jobs easier. Finally, we introduced artifact server, to keep the artifacts created in each stage of a build pipeline. Users and downstream developers can fetch these files for doing further testing. We will go through the history of FreeBSD’s CI work, and discuss how we we setup the latest CI system of the FreeBSD project, in which new build triggered by changes in the version control system, then the system does tests and publishes results. We'll explain the underlaying mechanisms and how we use interesting Jenkins plugins or other tools to make it more useful.
 So today, this talk is about continuous integration of the free busy project and I'm Li Wenxu. So yeah, firstly, something about me. I'm from Taiwan and my Chinese name is Xu Liwen. And I'm from Taiwan, as known as Formosa. I'm a system administrator. I'm an instructor to teach system and network administration at the college. And I'm a programmer. I'm also a traveler, school bus diver. And in the winter, I do snowboarding. And in the spring and autumn, I ride road bikes. So it takes about 15, more than 15 hours to get from Taipei to Ottawa. Yeah, a long trip. So my first DSD can, which is in 2014, the keynote said, yeah. Yeah, thanks for all of the developers. We have a rigorous engineering process. Yeah, that's wonderful. So but there shouldn't be a ceiling. So we always need a more higher quality of free BSD. So what do we want? We want a higher quality release. And we want the current branch is always buildable. We want stable branches always reliable. And besides that, we want the development process being faster and faster. Yeah, so in software development, there is a more state need to release early, release often. How to do that is trying to test early and the test often. So integration test is very important, not only for unit tests. So something like this. Unit test is not enough because we still need to do integration test or something just failed. Yeah, so we need to do integration test continuously. So in 90s, Grandi Buche invent or study using continuous integration CI this turn. And so it is just a software engineering practice of merging all developer working copies to a shared, manualized several times a day. So the reason is that we can know the conflicts and the know the bugs earlier as early as possible. So Wikipedia, it summarize some best practice of continuous integration. Yeah, so let's look inside those items. Firstly, we do this from the beginning, we have the version of the PC.org. So yeah, a good start. Then because we are an open source OS project, so there are some items not very applicable here. Just go see that. And so what we need is we need an automatic build. We should make each build a self-testing and every commit to a main trunk should be built. And we want every build is faster. And it is easy to get the latest results from each build. And everyone can see the results of the build. So not only the artifact created by build but also the status, the warnings or anything from the build. So that divides into two parts. First, the four items which can be provided by a continuous integration system. And the last two items which need help from everybody. So yeah, we need everybody to work on to make each commit test before commit. And then we need some more work to make our build faster. There are still many parts of our system cannot be built thoroughly. So there are still many ones to improve. So firstly, automatic build and sign history of our continuous integration. Yeah, start from 2002 to 2014, there is a service called Tintabox which is built by Das. The URL is Tintabox.vb.org. It probably started in March 2002, I dig into our mail archives. So first mail from Tintabox can be checked back to it. And it probably stopped in September 2014 because the last mail from it and the web pages stopped there. So it is open sourced, all the code is available on our subversion. And it is well documented in the freebie documentation project and on wiki page. So it is how it looked like. See, it just stopped somewhere in September 2014. So in 2013, Craig started a group called Jenkins admin. And currently we have Baptiste, Brad, Bruce, Craig, me and Steve in this group. And we have some alumni, Julio, Steven and Tyler are hand. So first the working group was in BSD-CAN 2014. And the land stopped for a while but it resumed the last EuroBSD-CAN. And we also had the Dev Summit yesterday. So for more information, just check out the wiki page. And we also have to do this on the wiki page. Feel free to add something or just discuss with us. So we are using Jenkins currently. Jenkins is probably the most well known and the popular defector standard of the CI in the industry. So it's an open source automation server written in Java by CosK. And the website is Jenkins.io. So it's just a Java demon that automatically is a non-human part of the whole software development process. So we use it for continuous integration and it even can do continuous delivery. So while it works, we also use Jenkins to develop our web service into a production environment. So the best part of it is Jenkins can be extended by various plugins. It is the best part of it. So along with Jenkins admin group created, of course, we have Jenkins.vv.org, which is of course a Jenkins website. And it starts to check out our code and build it. So it's, at first, it periodically builds and tests free VAD source and documentation trees for all supported branches. And the architectures by then it builds MD64, IE386, and the Sparks 64. And we also have GCC build, which is leveraged to a chain. So we are using MD64 to close build MD64 itself. So, yeah, and most of the code is available at Free VADCI repository at Free VADC organization on GitHub. So this is how it looks like. And it starts from last year right after BSD can, I started a new site called the CIDA Free VADC.org under the help by Foundation. Foundation supports the hardware, which is also based on Jenkins. But I'm trying to improve and experiment some new ideas and features and trying to fix the annoying part we learned from Jenkins.freebz.org. One of the interesting part is we introduced a server called Artifact server. So because during each part, each phase of the pipeline, there are many object files or artifact files need to pass between those files. Right now we just upload everything onto a server and not only for passing those files between each phases, but also we can let people to download those files and the tests and do anything they want. So we introduced Artifact server, which will be discussed more later. We create some job template to try to make each build jobs more organized. And now everything is also on FreeBDCI repository on GitHub. So a basic build pipeline is like this. Of course, firstly we need to build a code. Then we create a test virtual machine image and then we run this image with Beehive to run the real test inside virtual machine. After the test ends, the virtual machine gets shut down and we extract the results from these disk images. And besides this, we also introduced two more jobs. One is linked to build the linked kernel, make sure more part of the kernel is tested and it is buildable. And we also have another job called Images, which is creating an ordinary virtual machine image for people to try the latest snapshot. So automatic build on Cfbd.org contains two jobs. One is build job and another is linked, as we mentioned in previous slide. So the script to execute LAN is under scripts slash build directory. One is build worker and kernel, the other is build kernel, linked, that is the edge. Honestly, nothing special, just passing those invariable variables and it's killed Mac. So about the self-test, we use KUYA, which is a testing framework for infrastructure software created by Julio, who is both NetBSD and the free BD developer. The repository is also on GitHub, Julio slash KUYA. It is well documented. You can follow the QuickStar Wiki page. Our Julio gave a nice tutorial at Hrpftkang2014, so you can just look at the materials layer. And this is what we run test. And where we run the test, we use Beehive, the BSD hypervisor, which is in free BD since 10.0. And on CIDA free BD.org, we have two jobs to create virtual machine images for Beehive. One is images and another is test VM job, which just build two kinds of virtual machines. So we say images build ordinary virtual machine images and test VM job just extend that virtual machine image with sound pre-installed KUYA and the test files. And we also configured to run tested while as soon as it boots. And for RISC-5 architecture, we run it in QMU. So putting those things together. Firstly, we create virtual machine images to load jobs. And we put install KUYA and the test files under user slash test. And we customize slash ETC slash RCDALOCO file to run test and generate test reports. And finally, shut down virtual machine. And everything is under script slash build directory. And besides of this, these scripts also extract the results from that disk image. So first script is building a test image and the second script run test.sh is the developer machine extract results. So how to access those test results? We worked with Julio and make KUYA can produce many non-human readable outputs. This is called JUnit format output, which is basically an XML file. And this is not human readable, but after it is powered by Jenkins, it can create a beautiful report. So under each job, each build, you can have a URL slash test report. So you can see the test results. From here, you can see three test files in previous build has been fixed. And 64 has been skipped. I don't know why. So this is how the result looks like. And for the failed test case, it also has a nice display. You can browse into each test results and see the error message and what's on the standard output. And we also keep track of the history of the test results. So you can see the number of the test cases and the status of the test in each build. And for artifacts, which is generated in each phase of the pipeline, we have three kinds of artifacts for now. First one is the disk files, which is similar or which is identical to what we have on official FTP server. So those are base or kernel.htx files. So you can just grab them and install to test. And we also use those distribution files to create disk images for the virtual machine or the test virtual machine. So other downstream phase in the pipeline is just executing those virtual machine images. And for external users, you can just download those files to do analysis, to debug or to follow testers. So artifact.cfbc.org has two interfaces. One is used for those Jenkins Java to upload artifacts. So we page the Jenkins plugin to make it support TOS transmission for security reasons. And another is HTTP interface for downstream jobs to download artifacts from the upstream. And external users can also use this interface to download whatever they want. So open configurations. We want our state-double can be reproduced by others. So we want to make, and another goal is we want to make jobs creation easier because we find that not everyone knows and wants to know how to configure Jenkins, which is somehow tedious. And we also find that those build jobs have most parts in common, just differ in some arguments. So we create a job template. A job template, those three things, set up in building environment, execute the specific build script for that job, and finally it's clean up the environment. So for environment stand-up, which also contains three steps. First is check out the latest source code, then set up Nidia's version of FreeBSD. So we try to run every job inside the jail and the clean created jail to make sure everything is beautiful from scratch and not be affected by previously leftovers. And we also need to, inside that jail we also install the required packages for the spatial need of that job. And, but, AD4 Jenkins configuration can be only done by its web interface and the configuration files is stored as XML files on the disk, which is not very easy to manipulate and maintain. And after some research we found that from OpenStake project, they created a project called the Jenkins Drop Builder. What it does is very simple. It defines the YAML file settings and they created a Python script to read those YAML files and configure Jenkins through its web service interface. So we can just check those YAML files in any source code control system we like and use Jenkins Drop Builder to manipulate the job configuration for update or create and delete. So currently every job is configured by using Jenkins Drop Builder YAML to configure and the files are all in the JDB directory in our FreeBSD CI repository. So basically a basic job definition is like this. This is the example of a scan build job and this is the big picture of what it looks like. And basically each line of it is a marker and the marker definition is in other parts of the YAML file. The most important part of this YAML file is in Jail because following we will introduce is in Jail script set which does the build job. So building Jail script set is under script Jail directory and what it does is it simplifies the clean environment setup. So basically a job execution contains three steps. Set out the environment, execute whatever you want inside Jail and clean up Jail. Inside Jail we installs to do by default. You can use root. You can execute your job as root and configure the environment in a limited way. So typical job configuration contains three files. First is the build.sh which is the mandatory. This is the main build script. So you just put everything you want to execute in that file. The other two files is Jail.conf and the PKG list. Jail.conf is the configuration of the execution environment and the PKG list just contains every package you want to use inside Jail. So after we create this template adding a new job becomes very easy. So submitters just bring their own build scripts and the configurations. So those three files or just build.sh to us. If other configuration files is needed just bring everything to us such as source.conf or make.conf. Ideally it can be submit as you have a pro request. While we Jenkins and Minters just merge your job into the job directory. And we create a new entry in Jenkins Job Rebuilder. YAML files and the proceed to the production Jenkins. Then it's done. Your Jenkins Job Rebuilder is queued by Undyman. So this is an example of the Un64 build. So for Jail.conf we need ND64 as the Jail environment and 32-bit support is not necessary and the Jail version is 11.0. For PKG list we just put on ARC64 being utl. I know that it is not needed anymore but it's an example to show how to install or require the packages. And build.sh just execute make a word because we extract the commands of a build word and kernel into an initial script. So building Un64 is just passing the necessary arguments to that script. And besides this, ordinary jobs we also make job build this execute in Jail job state support a more called quarantine mode because sometimes we need to build our execute commands from external which means last chest bolt sources. So specify quarantine equals 1 to Jail.conf. What we do is make a Jail and isolate the environment. So after install all the requirement packages we just reconfigure the Jail and remove internet connection or other external dependencies. And we also limited the job resource from job configuration more strictly. First of all, this is job execution time we just configured to limit for 90 minutes. And after 90 minutes there will be an absolute fail or occurs to stop everything. So about other integrations of our CI environment. So first thing is notifications. Notification content which is very simple but also very hard because we need to determine how, when and what to not fight. And basically notify occurrence when better things happened but also when better things end since it got fixed we also send notification to tell everybody everything is fine. And how to notify. It is not turned on CI.vp.org but after we have more confidence with this we will use email notification and IRC to tell developers better things happened. And what to notify. We notify people about the changes since last build it to let people to figure out what changes makes better things. And the tail of the console output to more easy to get an idea and as for builder. And for test job we just give people the length and output of those cases. So email notification is somehow annoying if we can not very carefully. So this is the screenshot of last year. You see current building days to most of the senders are janky and mean. So some kind of annoying. Because there are no annoying probably nobody reads it and if nobody reads it it is totally not used. So we need to reduce false partives. And we found an email extension plugin which can send notification. And it gives some scripts before notification is sent. So we read some groovy school to add some patterns that is mostly patterns are janky internal errors. So we just fill out those false partives before sending emails to everyone. And for the contents we also use email extension plugin to customize mail body. So the template for failed or unstable build jobs is like this. Of course there is a title and then we provide necessary UI information to users. And some changes some are in the build log. So an example is look like this. This job fails and the necessary URLs changes some list and the output of the build log. And for failed or unstable test cases basically are the same. At the end we are not just telling the build log but also give the details of the failed cases. So things looks like this. There are two test cases failed. So following is the test case none and the error messages. And another thing we have done is integrate with our code review system, FabCator. So we have a job to test the pages from FabCator. Which is done by FabCator differential parking. It is first created by Uber but at first it supports only Git. And we page it to support subversion which we are using. And it gets upstreamed a few months ago. So that build job is like this. So we still have many builds is killed. And next to each build there is a differential number attached with it. And if we click to build page it will contains the detail of the differential information which is from FabCator. So yeah, this is an example. So at the FabCator site it also has the build status displayed on each differential. So after the job is complete, that build status will become green light or red light means successful or failed. And we also created a job using ClientScanBuilder to perform statical analyzing. So it is basically using a scan build from ClientProject. And we also use BSD.CleanAnalyze.mk which is imported from NetBSD. So we just skewed it and specify the output directory and specify we need HTML output. So the reason is like this. The type of the problem and which file has problems and the pass and the descriptions. So ClientScanBuilder produces a very detailed output of each problem it found. So people can go through those files and try to improve our code quality. And that is for our ST codes and for our documentation thanks to Warren who wrote a tool called Eager which can do a link of our main page or XML file, documentation files. And because my big job is Java programmer and we use checkstyle to check the Java files. So I and Warren came up with an idea. Why not just page Eager to produce a checkstyle output? Then we can use the Jenkins checkstyle plugin to display those warnings. So the reason is like this. For the documentation tree we not only build documentation tree to make sure it is buildable. We also create another Java to skew the Eager on each documentation file and provide a warning or mostly warning resource for people to improve the documentation quality. So there is also a history trend to let us know what seems getting better and better or worse and worse. And not only for state-run analysis and the quality of the documentation. We also do in the build we also collect warnings, compiler warnings from Cranel or GCC. So it is basically the same very similar output. So we have a file and a severity of the warning. And we also have trend to keep track of the history. And finally we use build status page to let people can embed the build status to other websites. So for R64 we give page we have the head and stable 11 build status. And which is also used by the riskify org. And under the freebd page of riskify page there is also a page status. The current freebd had a building state of riskify. So other Jenkins Java we use internally is we first we use SCM configuration to keep track of the... There are some system-wide global Jenkins configuration which is not controlled by Jenkins Java builder. And some of it then even contains sensitive data which cannot be published to the public. So we just put those configurations into private source code management repository. And we also use set restart to make sure we only restart Jenkins while knowing other jobs are running to prevent false positives. And finally some people suggest us to install green boards because Jenkins by default it uses blue light to indicate job successful. But green is more people like more intuitive to people. So yeah, everything is fine. And finally I would say Jenkins. We also use some leverage Jenkins REST API. So for almost every URL you can append the slash API to get the REST API documentation. So you can retrieve information from our server. So yeah, this is an example to get latest good to build revision of freebd. So you can just scroll to retrieve this JSON output and pass to JQ and get the latest buildable version revision. And by use that we thanks to the foundation intern Shiva and we create a similar view with Tinder box. So you can see the current status of each architecture and the each branches. Yes, currently our 32 bits on head is broken. Yeah, for now. So finally summary. Continuous integration does really help the development of freebd project. For the past year I keep poking developers to fix the code which they just forgot to test. And we accumulated much experience about that. So we hope those experience and the code can also help other projects. Yeah, there are always lots of the future work. Firstly, I want everybody can join the help to test freebd. We are on freebd testing mailing list and on EFnet list freebdci channel. And in the near future we want to use this mechanism to check for the produce build and we want to have more testers. Even sound testers are not enabled by default. So such as test for details. And we also want to build for the project branches for people more easy to test new experimental branches in project branch. Because in previously people can only check out by itself and build everything. If we also connect selected branches to our CI system we can also produce artifacts via images for those branches and make other people more easy to test. And thanks to Steve Wells he started to work on CI for ports. So we are trying to revive or trying to do something like our previous reports. And from the port management team they also want we can help the XP run process to make it more automatically. And Brad who is working on the network performance testers. And finally we also want to work with other projects such as other BSDs or other downstream projects to make those ecosystem more complete. So finally thanks to the colleges on Jenkins administration team and thanks for the foundation to buy us lots of machines to do the work. And thanks for the class admin to help us to set up those machines. And thanks for the fabricator admin to work with the fabricator integration. And thanks everyone on testing current stable mailing list to help the test project. Okay that's all. And happy to text our questions. Thanks. Yes. Excuse me. Sorry I don't really understand. Intentions I will always be proud of you. I'm learn a lot about perché and still wanting to built a good network these days to be great. in what branch? What branch? The DRM branch. DRM branch. Is there any test in particular that exists that would actually help test that branch? So it helps to test that we have more about the whole thing. Just regression testing. Yeah. Okay. Well, if you really need that, and I can quickly to create a job and which just check out, because we can just switch the source control system of large Java from subversion to whatever, of course, Git to, and then check out the, specify the branch and run the same test or just build for it. Yeah. Yeah. Of course, it's doable. Yeah, I don't know. If it's of much use. Well, the only limitation is the computing power. So if it is really needed, we can try to make a foundation to buy more hardware for us. Yeah. Okay. Yes. Okay. Yes. If I just want to build a single test VM test image like you do that just boots, runs the tests, that's it. What is the minimal set of things that I need to do? Is it just like a script that I run from your free DSDCI repository or are there changes in the build system that I can make test image or something? If you just want to build an ordinary VM image, there are some scripts on the release directory. And if you want to have test scripts and test files building, yeah, just run the scripts on the free DCI scripts. Okay. Yeah. So that will build a VM which boots automatically runs the complete test suite and then I shut down. Yes. It will be shut down automatically. Right. Yeah. Okay. So, yes. About how many branches are you covering right now? Right now, all the supported branches. So, yeah. This is all the branches and the architectures we test for now. We definitely like to extend to other supported tier one to tier three architectures but landing some more work and some more hardware resources. And roughly how many people are committing it, like commit rate average in these branches? How many commits per day in each of these branches? Yeah, I'm not really sure. Probably need to dig into the establishment history. Yeah. I saw the currently we produce probably 30 to 50 snapshots built per day. Yeah. So, any other questions? Just a quick question like that. How do you integrate with Pudria? How do you work with Pudria? Did you actually try to build it with Pudria? Pudria? The package builder? Yeah. Currently, no. And Steve has tried it but which is not very practical and we are trying to improve this. Yeah, we followed Dev Summit yesterday. We have some new ideas and we'll be trying in the following weeks. So, any other? Oh, yeah. So, Pudria is actually able to provide hardware that's suitable for this and that's why we are running on Java and the build pipeline being an issue if you wanted to do native builds or something like that. Or... How do we build RISC 5 and the R64? Well, yeah, so you said you use KMU. Yes. And that's driven off of X86 but I think, is there ever a point where you'll be running the CI on native hardware or RISC 5 native? Let me repeat your question. Are you asking about is it possible to run architectures like RISC 5 or R64 on native machines, native hardware? No, it was more on the lines of... So, everything's running on X86 and then it's spawned, right? So, the spawner machine, you ever see that being like RMOR or RISC 5 in the future? Is that desirable or is that...? To run test on a physical machine? Yes. Yeah, it is possible and which... In fact, at least it is on our roadmap but this requires some more spatial hardware. We need to probably need JTAG also to automatically install a new image on that board and collect the console output and maybe another power controller to rebuild that board. Yeah. Okay. So, in that case, the Jenkins Java worker on the X86 platform... So, before the slave, you can just use a Sysage, you know, like we... You just spoke about the Sysage. Yeah, the build slave can still be a X86 but we just connect those development board to that machine and last slave just is queued, commenced through the connection of that board. That's it. So, since no question now, thanks everybody again. Yeah. Thank you.
 So today, this talk is about continuous integration of the free busy project and I'm Li Wenxu.  So yeah, firstly, something about me.  I'm from Taiwan and my Chinese name is Xu Liwen.  And I'm from Taiwan, as known as Formosa.  I'm a system administrator. I'm an instructor to teach system and network administration at the college.  And I'm a programmer. I'm also a traveler, school bus diver.  And in the winter, I do snowboarding.  And in the spring and autumn, I ride road bikes.  So it takes about 15, more than 15 hours to get from Taipei to Ottawa.  Yeah, a long trip.  So my first DSD can, which is in 2014, the keynote said, yeah.  Yeah, thanks for all of the developers.  We have a rigorous engineering process.  Yeah, that's wonderful.  So but there shouldn't be a ceiling.  So we always need a more higher quality of free BSD.  So what do we want?  We want a higher quality release.  And we want the current branch is always buildable.  We want stable branches always reliable.  And besides that, we want the development process being faster and faster.  Yeah, so in software development, there is a more state need to release early, release often.  How to do that is trying to test early and the test often.  So integration test is very important, not only for unit tests.  So something like this.  Unit test is not enough because we still need to do integration test or something just failed.  Yeah, so we need to do integration test continuously.  So in 90s, Grandi Buche invent or study using continuous integration CI this turn.  And so it is just a software engineering practice of merging all developer working copies  to a shared, manualized several times a day.  So the reason is that we can know the conflicts and the know the bugs earlier as early as possible.  So Wikipedia, it summarize some best practice of continuous integration.  Yeah, so let's look inside those items.  Firstly, we do this from the beginning, we have the version of the PC.org.  So yeah, a good start.  Then because we are an open source OS project, so there are some items not very applicable here.  Just go see that.  And so what we need is we need an automatic build.  We should make each build a self-testing and every commit to a main trunk should be built.  And we want every build is faster.  And it is easy to get the latest results from each build.  And everyone can see the results of the build.  So not only the artifact created by build but also the status, the warnings or anything from the build.  So that divides into two parts.  First, the four items which can be provided by a continuous integration system.  And the last two items which need help from everybody.  So yeah, we need everybody to work on to make each commit test before commit.  And then we need some more work to make our build faster.  There are still many parts of our system cannot be built thoroughly.  So there are still many ones to improve.  So firstly, automatic build and sign history of our continuous integration.  Yeah, start from 2002 to 2014, there is a service called Tintabox which is built by Das.  The URL is Tintabox.vb.org.  It probably started in March 2002, I dig into our mail archives.  So first mail from Tintabox can be checked back to it.  And it probably stopped in September 2014 because the last mail from it and the web pages stopped there.  So it is open sourced, all the code is available on our subversion.  And it is well documented in the freebie documentation project and on wiki page.  So it is how it looked like.  See, it just stopped somewhere in September 2014.  So in 2013, Craig started a group called Jenkins admin.  And currently we have Baptiste, Brad, Bruce, Craig, me and Steve in this group.  And we have some alumni, Julio, Steven and Tyler are hand.  So first the working group was in BSD-CAN 2014.  And the land stopped for a while but it resumed the last EuroBSD-CAN.  And we also had the Dev Summit yesterday.  So for more information, just check out the wiki page.  And we also have to do this on the wiki page.  Feel free to add something or just discuss with us.  So we are using Jenkins currently.  Jenkins is probably the most well known and the popular defector standard of the CI in the industry.  So it's an open source automation server written in Java by CosK.  And the website is Jenkins.io.  So it's just a Java demon that automatically is a non-human part of the whole software development process.  So we use it for continuous integration and it even can do continuous delivery.  So while it works, we also use Jenkins to develop our web service into a production environment.  So the best part of it is Jenkins can be extended by various plugins.  It is the best part of it.  So along with Jenkins admin group created, of course, we have Jenkins.vv.org, which is of course a Jenkins website.  And it starts to check out our code and build it.  So it's, at first, it periodically builds and tests free VAD source and documentation trees for all supported branches.  And the architectures by then it builds MD64, IE386, and the Sparks 64.  And we also have GCC build, which is leveraged to a chain.  So we are using MD64 to close build MD64 itself.  So, yeah, and most of the code is available at Free VADCI repository at Free VADC organization on GitHub.  So this is how it looks like.  And it starts from last year right after BSD can, I started a new site called the CIDA Free VADC.org under the help by Foundation.  Foundation supports the hardware, which is also based on Jenkins.  But I'm trying to improve and experiment some new ideas and features and trying to fix the annoying part we learned from Jenkins.freebz.org.  One of the interesting part is we introduced a server called Artifact server.  So because during each part, each phase of the pipeline, there are many object files or artifact files need to pass between those files.  Right now we just upload everything onto a server and not only for passing those files between each phases,  but also we can let people to download those files and the tests and do anything they want.  So we introduced Artifact server, which will be discussed more later.  We create some job template to try to make each build jobs more organized.  And now everything is also on FreeBDCI repository on GitHub.  So a basic build pipeline is like this.  Of course, firstly we need to build a code.  Then we create a test virtual machine image and then we run this image with Beehive to run the real test inside virtual machine.  After the test ends, the virtual machine gets shut down and we extract the results from these disk images.  And besides this, we also introduced two more jobs.  One is linked to build the linked kernel, make sure more part of the kernel is tested and it is buildable.  And we also have another job called Images, which is creating an ordinary virtual machine image for people to try the latest snapshot.  So automatic build on Cfbd.org contains two jobs.  One is build job and another is linked, as we mentioned in previous slide.  So the script to execute LAN is under scripts slash build directory. One is build worker and kernel, the other is build kernel, linked, that is the edge.  Honestly, nothing special, just passing those invariable variables and it's killed Mac.  So about the self-test, we use KUYA, which is a testing framework for infrastructure software created by Julio, who is both NetBSD and the free BD developer.  The repository is also on GitHub, Julio slash KUYA. It is well documented.  You can follow the QuickStar Wiki page.  Our Julio gave a nice tutorial at Hrpftkang2014, so you can just look at the materials layer.  And this is what we run test.  And where we run the test, we use Beehive, the BSD hypervisor, which is in free BD since 10.0.  And on CIDA free BD.org, we have two jobs to create virtual machine images for Beehive. One is images and another is test VM job, which just build two kinds of virtual machines.  So we say images build ordinary virtual machine images and test VM job just extend that virtual machine image with sound pre-installed KUYA and the test files.  And we also configured to run tested while as soon as it boots.  And for RISC-5 architecture, we run it in QMU.  So putting those things together.  Firstly, we create virtual machine images to load jobs.  And we put install KUYA and the test files under user slash test. And we customize slash ETC slash RCDALOCO file to run test and generate test reports.  And finally, shut down virtual machine.  And everything is under script slash build directory.  And besides of this, these scripts also extract the results from that disk image.  So first script is building a test image and the second script run test.sh is the developer machine extract results.  So how to access those test results?  We worked with Julio and make KUYA can produce many non-human readable outputs.  This is called JUnit format output, which is basically an XML file.  And this is not human readable, but after it is powered by Jenkins, it can create a beautiful report.  So under each job, each build, you can have a URL slash test report.  So you can see the test results.  From here, you can see three test files in previous build has been fixed.  And 64 has been skipped.  I don't know why.  So this is how the result looks like.  And for the failed test case, it also has a nice display.  You can browse into each test results and see the error message and what's on the standard output.  And we also keep track of the history of the test results.  So you can see the number of the test cases and the status of the test in each build.  And for artifacts, which is generated in each phase of the pipeline, we have three kinds of artifacts for now.  First one is the disk files, which is similar or which is identical to what we have on official FTP server.  So those are base or kernel.htx files.  So you can just grab them and install to test.  And we also use those distribution files to create disk images for the virtual machine or the test virtual machine.  So other downstream phase in the pipeline is just executing those virtual machine images.  And for external users, you can just download those files to do analysis, to debug or to follow testers.  So artifact.cfbc.org has two interfaces.  One is used for those Jenkins Java to upload artifacts.  So we page the Jenkins plugin to make it support TOS transmission for security reasons.  And another is HTTP interface for downstream jobs to download artifacts from the upstream.  And external users can also use this interface to download whatever they want.  So open configurations.  We want our state-double can be reproduced by others.  So we want to make, and another goal is we want to make jobs creation easier because we find that not everyone knows and wants to know how to configure Jenkins,  which is somehow tedious.  And we also find that those build jobs have most parts in common, just differ in some arguments.  So we create a job template.  A job template, those three things, set up in building environment, execute the specific build script for that job, and finally it's clean up the environment.  So for environment stand-up, which also contains three steps.  First is check out the latest source code, then set up Nidia's version of FreeBSD.  So we try to run every job inside the jail and the clean created jail to make sure everything is beautiful from scratch and not be affected by previously leftovers.  And we also need to, inside that jail we also install the required packages for the spatial need of that job.  And, but, AD4 Jenkins configuration can be only done by its web interface and the configuration files is stored as XML files on the disk,  which is not very easy to manipulate and maintain.  And after some research we found that from OpenStake project, they created a project called the Jenkins Drop Builder.  What it does is very simple.  It defines the YAML file settings and they created a Python script to read those YAML files and configure Jenkins through its web service interface.  So we can just check those YAML files in any source code control system we like and use Jenkins Drop Builder to manipulate the job configuration for update or create and delete.  So currently every job is configured by using Jenkins Drop Builder YAML to configure and the files are all in the JDB directory in our FreeBSD CI repository.  So basically a basic job definition is like this. This is the example of a scan build job and this is the big picture of what it looks like.  And basically each line of it is a marker and the marker definition is in other parts of the YAML file.  The most important part of this YAML file is in Jail because following we will introduce is in Jail script set which does the build job.  So building Jail script set is under script Jail directory and what it does is it simplifies the clean environment setup.  So basically a job execution contains three steps. Set out the environment, execute whatever you want inside Jail and clean up Jail.  Inside Jail we installs to do by default. You can use root. You can execute your job as root and configure the environment in a limited way.  So typical job configuration contains three files. First is the build.sh which is the mandatory. This is the main build script.  So you just put everything you want to execute in that file.  The other two files is Jail.conf and the PKG list. Jail.conf is the configuration of the execution environment and the PKG list just contains every package you want to use inside Jail.  So after we create this template adding a new job becomes very easy. So submitters just bring their own build scripts and the configurations.  So those three files or just build.sh to us. If other configuration files is needed just bring everything to us such as source.conf or make.conf.  Ideally it can be submit as you have a pro request. While we Jenkins and Minters just merge your job into the job directory.  And we create a new entry in Jenkins Job Rebuilder. YAML files and the proceed to the production Jenkins. Then it's done. Your Jenkins Job Rebuilder is queued by Undyman.  So this is an example of the Un64 build. So for Jail.conf we need ND64 as the Jail environment and 32-bit support is not necessary and the Jail version is 11.0.  For PKG list we just put on ARC64 being utl. I know that it is not needed anymore but it's an example to show how to install or require the packages.  And build.sh just execute make a word because we extract the commands of a build word and kernel into an initial script.  So building Un64 is just passing the necessary arguments to that script.  And besides this, ordinary jobs we also make job build this execute in Jail job state support a more called quarantine mode because sometimes we need to build our execute commands from external which means last chest bolt sources.  So specify quarantine equals 1 to Jail.conf. What we do is make a Jail and isolate the environment.  So after install all the requirement packages we just reconfigure the Jail and remove internet connection or other external dependencies.  And we also limited the job resource from job configuration more strictly.  First of all, this is job execution time we just configured to limit for 90 minutes.  And after 90 minutes there will be an absolute fail or occurs to stop everything.  So about other integrations of our CI environment.  So first thing is notifications. Notification content which is very simple but also very hard because we need to determine how, when and what to not fight.  And basically notify occurrence when better things happened but also when better things end since it got fixed we also send notification to tell everybody everything is fine.  And how to notify. It is not turned on CI.vp.org but after we have more confidence with this we will use email notification and IRC to tell developers better things happened.  And what to notify. We notify people about the changes since last build it to let people to figure out what changes makes better things.  And the tail of the console output to more easy to get an idea and as for builder.  And for test job we just give people the length and output of those cases.  So email notification is somehow annoying if we can not very carefully.  So this is the screenshot of last year. You see current building days to most of the senders are janky and mean.  So some kind of annoying.  Because there are no annoying probably nobody reads it and if nobody reads it it is totally not used. So we need to reduce false partives.  And we found an email extension plugin which can send notification.  And it gives some scripts before notification is sent. So we read some groovy school to add some patterns that is mostly patterns are janky internal errors.  So we just fill out those false partives before sending emails to everyone.  And for the contents we also use email extension plugin to customize mail body.  So the template for failed or unstable build jobs is like this.  Of course there is a title and then we provide necessary UI information to users.  And some changes some are in the build log.  So an example is look like this.  This job fails and the necessary URLs changes some list and the output of the build log.  And for failed or unstable test cases basically are the same.  At the end we are not just telling the build log but also give the details of the failed cases.  So things looks like this.  There are two test cases failed.  So following is the test case none and the error messages.  And another thing we have done is integrate with our code review system, FabCator.  So we have a job to test the pages from FabCator.  Which is done by FabCator differential parking. It is first created by Uber but at first it supports only Git.  And we page it to support subversion which we are using.  And it gets upstreamed a few months ago.  So that build job is like this.  So we still have many builds is killed.  And next to each build there is a differential number attached with it.  And if we click to build page it will contains the detail of the differential information which is from FabCator.  So yeah, this is an example.  So at the FabCator site it also has the build status displayed on each differential.  So after the job is complete, that build status will become green light or red light means successful or failed.  And we also created a job using ClientScanBuilder to perform statical analyzing.  So it is basically using a scan build from ClientProject.  And we also use BSD.CleanAnalyze.mk which is imported from NetBSD.  So we just skewed it and specify the output directory and specify we need HTML output.  So the reason is like this.  The type of the problem and which file has problems and the pass and the descriptions.  So ClientScanBuilder produces a very detailed output of each problem it found.  So people can go through those files and try to improve our code quality.  And that is for our ST codes and for our documentation thanks to Warren who wrote a tool called Eager  which can do a link of our main page or XML file, documentation files.  And because my big job is Java programmer and we use checkstyle to check the Java files.  So I and Warren came up with an idea.  Why not just page Eager to produce a checkstyle output?  Then we can use the Jenkins checkstyle plugin to display those warnings.  So the reason is like this.  For the documentation tree we not only build documentation tree to make sure it is buildable.  We also create another Java to skew the Eager on each documentation file  and provide a warning or mostly warning resource for people to improve the documentation quality.  So there is also a history trend to let us know what seems getting better and better or worse and worse.  And not only for state-run analysis and the quality of the documentation.  We also do in the build we also collect warnings, compiler warnings from Cranel or GCC.  So it is basically the same very similar output.  So we have a file and a severity of the warning.  And we also have trend to keep track of the history.  And finally we use build status page to let people can embed the build status to other websites.  So for R64 we give page we have the head and stable 11 build status.  And which is also used by the riskify org.  And under the freebd page of riskify page there is also a page status.  The current freebd had a building state of riskify.  So other Jenkins Java we use internally is we first we use SCM configuration to keep track of the...  There are some system-wide global Jenkins configuration which is not controlled by Jenkins Java builder.  And some of it then even contains sensitive data which cannot be published to the public.  So we just put those configurations into private source code management repository.  And we also use set restart to make sure we only restart Jenkins while knowing other jobs are running to prevent false positives.  And finally some people suggest us to install green boards because Jenkins by default it uses blue light to indicate job successful.  But green is more people like more intuitive to people.  So yeah, everything is fine.  And finally I would say Jenkins.  We also use some leverage Jenkins REST API.  So for almost every URL you can append the slash API to get the REST API documentation.  So you can retrieve information from our server.  So yeah, this is an example to get latest good to build revision of freebd.  So you can just scroll to retrieve this JSON output and pass to JQ and get the latest buildable version revision.  And by use that we thanks to the foundation intern Shiva and we create a similar view with Tinder box.  So you can see the current status of each architecture and the each branches.  Yes, currently our 32 bits on head is broken.  Yeah, for now.  So finally summary.  Continuous integration does really help the development of freebd project.  For the past year I keep poking developers to fix the code which they just forgot to test.  And we accumulated much experience about that.  So we hope those experience and the code can also help other projects.  Yeah, there are always lots of the future work.  Firstly, I want everybody can join the help to test freebd.  We are on freebd testing mailing list and on EFnet list freebdci channel.  And in the near future we want to use this mechanism to check for the produce build and we want to have more testers.  Even sound testers are not enabled by default.  So such as test for details.  And we also want to build for the project branches for people more easy to test new experimental branches in project branch.  Because in previously people can only check out by itself and build everything.  If we also connect selected branches to our CI system we can also produce artifacts via images for those branches and make other people more easy to test.  And thanks to Steve Wells he started to work on CI for ports.  So we are trying to revive or trying to do something like our previous reports.  And from the port management team they also want we can help the XP run process to make it more automatically.  And Brad who is working on the network performance testers.  And finally we also want to work with other projects such as other BSDs or other downstream projects to make those ecosystem more complete.  So finally thanks to the colleges on Jenkins administration team and thanks for the foundation to buy us lots of machines to do the work.  And thanks for the class admin to help us to set up those machines.  And thanks for the fabricator admin to work with the fabricator integration.  And thanks everyone on testing current stable mailing list to help the test project.  Okay that's all.  And happy to text our questions.  Thanks. Yes.  Excuse me.  Sorry I don't really understand.  Intentions I will always be proud of you.  I'm learn a lot about perché and still wanting to built a good network these days to be great.  in what branch?  What branch?  The DRM branch.  DRM branch.  Is there any test in particular that exists that would actually help test that branch?  So it helps to test that we have more about the whole thing.  Just regression testing.  Yeah.  Okay.  Well, if you really need that, and I can quickly to create a job and which just check out,  because we can just switch the source control system of large Java from subversion to whatever,  of course, Git to, and then check out the, specify the branch and run the same test or  just build for it.  Yeah.  Yeah.  Of course, it's doable.  Yeah, I don't know.  If it's of much use.  Well, the only limitation is the computing power.  So if it is really needed, we can try to make a foundation to buy more hardware for us.  Yeah.  Okay.  Yes.  Okay.  Yes.  If I just want to build a single test VM test image like you do that just boots, runs the  tests, that's it.  What is the minimal set of things that I need to do?  Is it just like a script that I run from your free DSDCI repository or are there changes  in the build system that I can make test image or something?  If you just want to build an ordinary VM image, there are some scripts on the release directory.  And if you want to have test scripts and test files building, yeah, just run the scripts  on the free DCI scripts.  Okay.  Yeah.  So that will build a VM which boots automatically runs the complete test suite and then I shut  down.  Yes.  It will be shut down automatically.  Right.  Yeah.  Okay.  So, yes.  About how many branches are you covering right now?  Right now, all the supported branches.  So, yeah.  This is all the branches and the architectures we test for now.  We definitely like to extend to other supported tier one to tier three architectures but landing  some more work and some more hardware resources.  And roughly how many people are committing it, like commit rate average in these branches?  How many commits per day in each of these branches?  Yeah, I'm not really sure.  Probably need to dig into the establishment history.  Yeah.  I saw the currently we produce probably 30 to 50 snapshots built per day.  Yeah.  So, any other questions?  Just a quick question like that.  How do you integrate with Pudria?  How do you work with Pudria?  Did you actually try to build it with Pudria?  Pudria?  The package builder?  Yeah.  Currently, no.  And Steve has tried it but which is not very practical and we are trying to improve this.  Yeah, we followed Dev Summit yesterday.  We have some new ideas and we'll be trying in the following weeks.  So, any other?  Oh, yeah.  So, Pudria is actually able to provide hardware that's suitable for this and that's why we  are running on Java and the build pipeline being an issue if you wanted to do native  builds or something like that.  Or...  How do we build RISC 5 and the R64?  Well, yeah, so you said you use KMU.  Yes.  And that's driven off of X86 but I think, is there ever a point where you'll be running  the CI on native hardware or RISC 5 native?  Let me repeat your question.  Are you asking about is it possible to run architectures like RISC 5 or R64 on native  machines, native hardware?  No, it was more on the lines of...  So, everything's running on X86 and then it's spawned, right?  So, the spawner machine, you ever see that being like RMOR or RISC 5 in the future?  Is that desirable or is that...?  To run test on a physical machine?  Yes.  Yeah, it is possible and which...  In fact, at least it is on our roadmap but this requires some more spatial hardware.  We need to probably need JTAG also to automatically install a new image on that board and collect  the console output and maybe another power controller to rebuild that board.  Yeah.  Okay.  So, in that case, the Jenkins Java worker on the X86 platform...  So, before the slave, you can just use a Sysage, you know, like we...  You just spoke about the Sysage.  Yeah, the build slave can still be a X86 but we just connect those development board to  that machine and last slave just is queued, commenced through the connection of that board.  That's it.  So, since no question now, thanks everybody again.  Yeah.  Thank you.
Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_0_1955.png Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_1956_2603.png Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_2604_2651.png Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_2652_2783.png Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_2784_2867.png Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_2868_2939.png Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_2940_3419.png Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_3420_3743.png Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_3744_4571.png Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_4572_4931.png Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_4932_5927.png Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_5928_5975.png Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_5976_7319.png Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_7320_7439.png Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_7440_18287.png Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_18288_19355.png Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_19356_20639.png Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_20640_21887.png Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_21888_23483.png Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_23484_23819.png Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_23820_23879.png Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_23880_29099.png Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_29100_29195.png Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_29196_29279.png Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_29280_30227.png Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_30228_32279.png Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_32280_33863.png Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_33864_37643.png Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_37644_38087.png Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_38088_39011.png Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_39012_39287.png Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_39288_42995.png Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_42996_43067.png Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_43068_43115.png Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_43116_43199.png Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_43200_47567.png Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_47568_47639.png Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_47640_53735.png Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_53736_57395.png Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_57396_59759.png Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_59760_60155.png Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_60156_60947.png Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_60948_63575.png Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_63576_73859.png Li-WenHsuContinuousIntegrationofTheFreeBSDProjectBSDCan2017_73860_81161.png
So today, this talk is about continuous integration of the free busy project and I'm Li Wenxu. So yeah, firstly, something about me. I'm from Taiwan and my Chinese name is Xu Liwen. And I'm from Taiwan, as known as Formosa. I'm a system administrator. I'm an instructor to teach system and network administration at the college. And I'm a programmer. I'm also a traveler, school bus diver. And in the winter, I do snowboarding. And in the spring and autumn, I ride road bikes. So it takes about 15, more than 15 hours to get from Taipei to Ottawa. Yeah, a long trip. So my first DSD can, which is in 2014, the keynote said, yeah. Yeah, thanks for all of the developers. We have a rigorous engineering process. Yeah, that's wonderful. So but there shouldn't be a ceiling. So we always need a more higher quality of free BSD. So what do we want? We want a higher quality release. And we want the current branch is always buildable. We want stable branches always reliable. And besides that, we want the development process being faster and faster. Yeah, so in software development, there is a more state need to release early, release often. How to do that is trying to test early and the test often. So integration test is very important, not only for unit tests. So something like this. Unit test is not enough because we still need to do integration test or something just failed. Yeah, so we need to do integration test continuously. So in 90s, Grandi Buche invent or study using continuous integration CI this turn. And so it is just a software engineering practice of merging all developer working copies to a shared, manualized several times a day. So the reason is that we can know the conflicts and the know the bugs earlier as early as possible. So Wikipedia, it summarize some best practice of continuous integration. Yeah, so let's look inside those items. Firstly, we do this from the beginning, we have the version of the PC.org. So yeah, a good start. Then because we are an open source OS project, so there are some items not very applicable here. Just go see that. And so what we need is we need an automatic build. We should make each build a self-testing and every commit to a main trunk should be built. And we want every build is faster. And it is easy to get the latest results from each build. And everyone can see the results of the build. So not only the artifact created by build but also the status, the warnings or anything from the build. So that divides into two parts. First, the four items which can be provided by a continuous integration system. And the last two items which need help from everybody. So yeah, we need everybody to work on to make each commit test before commit. And then we need some more work to make our build faster. There are still many parts of our system cannot be built thoroughly. So there are still many ones to improve. So firstly, automatic build and sign history of our continuous integration. Yeah, start from 2002 to 2014, there is a service called Tintabox which is built by Das. The URL is Tintabox.vb.org. It probably started in March 2002, I dig into our mail archives. So first mail from Tintabox can be checked back to it. And it probably stopped in September 2014 because the last mail from it and the web pages stopped there. So it is open sourced, all the code is available on our subversion. And it is well documented in the freebie documentation project and on wiki page. So it is how it looked like. See, it just stopped somewhere in September 2014. So in 2013, Craig started a group called Jenkins admin. And currently we have Baptiste, Brad, Bruce, Craig, me and Steve in this group. And we have some alumni, Julio, Steven and Tyler are hand. So first the working group was in BSD-CAN 2014. And the land stopped for a while but it resumed the last EuroBSD-CAN. And we also had the Dev Summit yesterday. So for more information, just check out the wiki page. And we also have to do this on the wiki page. Feel free to add something or just discuss with us. So we are using Jenkins currently. Jenkins is probably the most well known and the popular defector standard of the CI in the industry. So it's an open source automation server written in Java by CosK. And the website is Jenkins.io. So it's just a Java demon that automatically is a non-human part of the whole software development process. So we use it for continuous integration and it even can do continuous delivery. So while it works, we also use Jenkins to develop our web service into a production environment. So the best part of it is Jenkins can be extended by various plugins. It is the best part of it. So along with Jenkins admin group created, of course, we have Jenkins.vv.org, which is of course a Jenkins website. And it starts to check out our code and build it. So it's, at first, it periodically builds and tests free VAD source and documentation trees for all supported branches. And the architectures by then it builds MD64, IE386, and the Sparks 64. And we also have GCC build, which is leveraged to a chain. So we are using MD64 to close build MD64 itself. So, yeah, and most of the code is available at Free VADCI repository at Free VADC organization on GitHub. So this is how it looks like. And it starts from last year right after BSD can, I started a new site called the CIDA Free VADC.org under the help by Foundation. Foundation supports the hardware, which is also based on Jenkins. But I'm trying to improve and experiment some new ideas and features and trying to fix the annoying part we learned from Jenkins.freebz.org. One of the interesting part is we introduced a server called Artifact server. So because during each part, each phase of the pipeline, there are many object files or artifact files need to pass between those files. Right now we just upload everything onto a server and not only for passing those files between each phases, but also we can let people to download those files and the tests and do anything they want. So we introduced Artifact server, which will be discussed more later. We create some job template to try to make each build jobs more organized. And now everything is also on FreeBDCI repository on GitHub. So a basic build pipeline is like this. Of course, firstly we need to build a code. Then we create a test virtual machine image and then we run this image with Beehive to run the real test inside virtual machine. After the test ends, the virtual machine gets shut down and we extract the results from these disk images. And besides this, we also introduced two more jobs. One is linked to build the linked kernel, make sure more part of the kernel is tested and it is buildable. And we also have another job called Images, which is creating an ordinary virtual machine image for people to try the latest snapshot. So automatic build on Cfbd.org contains two jobs. One is build job and another is linked, as we mentioned in previous slide. So the script to execute LAN is under scripts slash build directory. One is build worker and kernel, the other is build kernel, linked, that is the edge. Honestly, nothing special, just passing those invariable variables and it's killed Mac. So about the self-test, we use KUYA, which is a testing framework for infrastructure software created by Julio, who is both NetBSD and the free BD developer. The repository is also on GitHub, Julio slash KUYA. It is well documented. You can follow the QuickStar Wiki page. Our Julio gave a nice tutorial at Hrpftkang2014, so you can just look at the materials layer. And this is what we run test. And where we run the test, we use Beehive, the BSD hypervisor, which is in free BD since 10.0. And on CIDA free BD.org, we have two jobs to create virtual machine images for Beehive. One is images and another is test VM job, which just build two kinds of virtual machines. So we say images build ordinary virtual machine images and test VM job just extend that virtual machine image with sound pre-installed KUYA and the test files. And we also configured to run tested while as soon as it boots. And for RISC-5 architecture, we run it in QMU. So putting those things together. Firstly, we create virtual machine images to load jobs. And we put install KUYA and the test files under user slash test. And we customize slash ETC slash RCDALOCO file to run test and generate test reports. And finally, shut down virtual machine. And everything is under script slash build directory. And besides of this, these scripts also extract the results from that disk image. So first script is building a test image and the second script run test.sh is the developer machine extract results. So how to access those test results? We worked with Julio and make KUYA can produce many non-human readable outputs. This is called JUnit format output, which is basically an XML file. And this is not human readable, but after it is powered by Jenkins, it can create a beautiful report. So under each job, each build, you can have a URL slash test report. So you can see the test results. From here, you can see three test files in previous build has been fixed. And 64 has been skipped. I don't know why. So this is how the result looks like. And for the failed test case, it also has a nice display. You can browse into each test results and see the error message and what's on the standard output. And we also keep track of the history of the test results. So you can see the number of the test cases and the status of the test in each build. And for artifacts, which is generated in each phase of the pipeline, we have three kinds of artifacts for now. First one is the disk files, which is similar or which is identical to what we have on official FTP server. So those are base or kernel.htx files. So you can just grab them and install to test. And we also use those distribution files to create disk images for the virtual machine or the test virtual machine. So other downstream phase in the pipeline is just executing those virtual machine images. And for external users, you can just download those files to do analysis, to debug or to follow testers. So artifact.cfbc.org has two interfaces. One is used for those Jenkins Java to upload artifacts. So we page the Jenkins plugin to make it support TOS transmission for security reasons. And another is HTTP interface for downstream jobs to download artifacts from the upstream. And external users can also use this interface to download whatever they want. So open configurations. We want our state-double can be reproduced by others. So we want to make, and another goal is we want to make jobs creation easier because we find that not everyone knows and wants to know how to configure Jenkins, which is somehow tedious. And we also find that those build jobs have most parts in common, just differ in some arguments. So we create a job template. A job template, those three things, set up in building environment, execute the specific build script for that job, and finally it's clean up the environment. So for environment stand-up, which also contains three steps. First is check out the latest source code, then set up Nidia's version of FreeBSD. So we try to run every job inside the jail and the clean created jail to make sure everything is beautiful from scratch and not be affected by previously leftovers. And we also need to, inside that jail we also install the required packages for the spatial need of that job. And, but, AD4 Jenkins configuration can be only done by its web interface and the configuration files is stored as XML files on the disk, which is not very easy to manipulate and maintain. And after some research we found that from OpenStake project, they created a project called the Jenkins Drop Builder. What it does is very simple. It defines the YAML file settings and they created a Python script to read those YAML files and configure Jenkins through its web service interface. So we can just check those YAML files in any source code control system we like and use Jenkins Drop Builder to manipulate the job configuration for update or create and delete. So currently every job is configured by using Jenkins Drop Builder YAML to configure and the files are all in the JDB directory in our FreeBSD CI repository. So basically a basic job definition is like this. This is the example of a scan build job and this is the big picture of what it looks like. And basically each line of it is a marker and the marker definition is in other parts of the YAML file. The most important part of this YAML file is in Jail because following we will introduce is in Jail script set which does the build job. So building Jail script set is under script Jail directory and what it does is it simplifies the clean environment setup. So basically a job execution contains three steps. Set out the environment, execute whatever you want inside Jail and clean up Jail. Inside Jail we installs to do by default. You can use root. You can execute your job as root and configure the environment in a limited way. So typical job configuration contains three files. First is the build.sh which is the mandatory. This is the main build script. So you just put everything you want to execute in that file. The other two files is Jail.conf and the PKG list. Jail.conf is the configuration of the execution environment and the PKG list just contains every package you want to use inside Jail. So after we create this template adding a new job becomes very easy. So submitters just bring their own build scripts and the configurations. So those three files or just build.sh to us. If other configuration files is needed just bring everything to us such as source.conf or make.conf. Ideally it can be submit as you have a pro request. While we Jenkins and Minters just merge your job into the job directory. And we create a new entry in Jenkins Job Rebuilder. YAML files and the proceed to the production Jenkins. Then it's done. Your Jenkins Job Rebuilder is queued by Undyman. So this is an example of the Un64 build. So for Jail.conf we need ND64 as the Jail environment and 32-bit support is not necessary and the Jail version is 11.0. For PKG list we just put on ARC64 being utl. I know that it is not needed anymore but it's an example to show how to install or require the packages. And build.sh just execute make a word because we extract the commands of a build word and kernel into an initial script. So building Un64 is just passing the necessary arguments to that script. And besides this, ordinary jobs we also make job build this execute in Jail job state support a more called quarantine mode because sometimes we need to build our execute commands from external which means last chest bolt sources. So specify quarantine equals 1 to Jail.conf. What we do is make a Jail and isolate the environment. So after install all the requirement packages we just reconfigure the Jail and remove internet connection or other external dependencies. And we also limited the job resource from job configuration more strictly. First of all, this is job execution time we just configured to limit for 90 minutes. And after 90 minutes there will be an absolute fail or occurs to stop everything. So about other integrations of our CI environment. So first thing is notifications. Notification content which is very simple but also very hard because we need to determine how, when and what to not fight. And basically notify occurrence when better things happened but also when better things end since it got fixed we also send notification to tell everybody everything is fine. And how to notify. It is not turned on CI.vp.org but after we have more confidence with this we will use email notification and IRC to tell developers better things happened. And what to notify. We notify people about the changes since last build it to let people to figure out what changes makes better things. And the tail of the console output to more easy to get an idea and as for builder. And for test job we just give people the length and output of those cases. So email notification is somehow annoying if we can not very carefully. So this is the screenshot of last year. You see current building days to most of the senders are janky and mean. So some kind of annoying. Because there are no annoying probably nobody reads it and if nobody reads it it is totally not used. So we need to reduce false partives. And we found an email extension plugin which can send notification. And it gives some scripts before notification is sent. So we read some groovy school to add some patterns that is mostly patterns are janky internal errors. So we just fill out those false partives before sending emails to everyone. And for the contents we also use email extension plugin to customize mail body. So the template for failed or unstable build jobs is like this. Of course there is a title and then we provide necessary UI information to users. And some changes some are in the build log. So an example is look like this. This job fails and the necessary URLs changes some list and the output of the build log. And for failed or unstable test cases basically are the same. At the end we are not just telling the build log but also give the details of the failed cases. So things looks like this. There are two test cases failed. So following is the test case none and the error messages. And another thing we have done is integrate with our code review system, FabCator. So we have a job to test the pages from FabCator. Which is done by FabCator differential parking. It is first created by Uber but at first it supports only Git. And we page it to support subversion which we are using. And it gets upstreamed a few months ago. So that build job is like this. So we still have many builds is killed. And next to each build there is a differential number attached with it. And if we click to build page it will contains the detail of the differential information which is from FabCator. So yeah, this is an example. So at the FabCator site it also has the build status displayed on each differential. So after the job is complete, that build status will become green light or red light means successful or failed. And we also created a job using ClientScanBuilder to perform statical analyzing. So it is basically using a scan build from ClientProject. And we also use BSD.CleanAnalyze.mk which is imported from NetBSD. So we just skewed it and specify the output directory and specify we need HTML output. So the reason is like this. The type of the problem and which file has problems and the pass and the descriptions. So ClientScanBuilder produces a very detailed output of each problem it found. So people can go through those files and try to improve our code quality. And that is for our ST codes and for our documentation thanks to Warren who wrote a tool called Eager which can do a link of our main page or XML file, documentation files. And because my big job is Java programmer and we use checkstyle to check the Java files. So I and Warren came up with an idea. Why not just page Eager to produce a checkstyle output? Then we can use the Jenkins checkstyle plugin to display those warnings. So the reason is like this. For the documentation tree we not only build documentation tree to make sure it is buildable. We also create another Java to skew the Eager on each documentation file and provide a warning or mostly warning resource for people to improve the documentation quality. So there is also a history trend to let us know what seems getting better and better or worse and worse. And not only for state-run analysis and the quality of the documentation. We also do in the build we also collect warnings, compiler warnings from Cranel or GCC. So it is basically the same very similar output. So we have a file and a severity of the warning. And we also have trend to keep track of the history. And finally we use build status page to let people can embed the build status to other websites. So for R64 we give page we have the head and stable 11 build status. And which is also used by the riskify org. And under the freebd page of riskify page there is also a page status. The current freebd had a building state of riskify. So other Jenkins Java we use internally is we first we use SCM configuration to keep track of the... There are some system-wide global Jenkins configuration which is not controlled by Jenkins Java builder. And some of it then even contains sensitive data which cannot be published to the public. So we just put those configurations into private source code management repository. And we also use set restart to make sure we only restart Jenkins while knowing other jobs are running to prevent false positives. And finally some people suggest us to install green boards because Jenkins by default it uses blue light to indicate job successful. But green is more people like more intuitive to people. So yeah, everything is fine. And finally I would say Jenkins. We also use some leverage Jenkins REST API. So for almost every URL you can append the slash API to get the REST API documentation. So you can retrieve information from our server. So yeah, this is an example to get latest good to build revision of freebd. So you can just scroll to retrieve this JSON output and pass to JQ and get the latest buildable version revision. And by use that we thanks to the foundation intern Shiva and we create a similar view with Tinder box. So you can see the current status of each architecture and the each branches. Yes, currently our 32 bits on head is broken. Yeah, for now. So finally summary. Continuous integration does really help the development of freebd project. For the past year I keep poking developers to fix the code which they just forgot to test. And we accumulated much experience about that. So we hope those experience and the code can also help other projects. Yeah, there are always lots of the future work. Firstly, I want everybody can join the help to test freebd. We are on freebd testing mailing list and on EFnet list freebdci channel. And in the near future we want to use this mechanism to check for the produce build and we want to have more testers. Even sound testers are not enabled by default. So such as test for details. And we also want to build for the project branches for people more easy to test new experimental branches in project branch. Because in previously people can only check out by itself and build everything. If we also connect selected branches to our CI system we can also produce artifacts via images for those branches and make other people more easy to test. And thanks to Steve Wells he started to work on CI for ports. So we are trying to revive or trying to do something like our previous reports. And from the port management team they also want we can help the XP run process to make it more automatically. And Brad who is working on the network performance testers. And finally we also want to work with other projects such as other BSDs or other downstream projects to make those ecosystem more complete. So finally thanks to the colleges on Jenkins administration team and thanks for the foundation to buy us lots of machines to do the work. And thanks for the class admin to help us to set up those machines. And thanks for the fabricator admin to work with the fabricator integration. And thanks everyone on testing current stable mailing list to help the test project. Okay that's all. And happy to text our questions. Thanks. Yes. Excuse me. Sorry I don't really understand. Intentions I will always be proud of you. I'm learn a lot about perché and still wanting to built a good network these days to be great. in what branch? What branch? The DRM branch. DRM branch. Is there any test in particular that exists that would actually help test that branch? So it helps to test that we have more about the whole thing. Just regression testing. Yeah. Okay. Well, if you really need that, and I can quickly to create a job and which just check out, because we can just switch the source control system of large Java from subversion to whatever, of course, Git to, and then check out the, specify the branch and run the same test or just build for it. Yeah. Yeah. Of course, it's doable. Yeah, I don't know. If it's of much use. Well, the only limitation is the computing power. So if it is really needed, we can try to make a foundation to buy more hardware for us. Yeah. Okay. Yes. Okay. Yes. If I just want to build a single test VM test image like you do that just boots, runs the tests, that's it. What is the minimal set of things that I need to do? Is it just like a script that I run from your free DSDCI repository or are there changes in the build system that I can make test image or something? If you just want to build an ordinary VM image, there are some scripts on the release directory. And if you want to have test scripts and test files building, yeah, just run the scripts on the free DCI scripts. Okay. Yeah. So that will build a VM which boots automatically runs the complete test suite and then I shut down. Yes. It will be shut down automatically. Right. Yeah. Okay. So, yes. About how many branches are you covering right now? Right now, all the supported branches. So, yeah. This is all the branches and the architectures we test for now. We definitely like to extend to other supported tier one to tier three architectures but landing some more work and some more hardware resources. And roughly how many people are committing it, like commit rate average in these branches? How many commits per day in each of these branches? Yeah, I'm not really sure. Probably need to dig into the establishment history. Yeah. I saw the currently we produce probably 30 to 50 snapshots built per day. Yeah. So, any other questions? Just a quick question like that. How do you integrate with Pudria? How do you work with Pudria? Did you actually try to build it with Pudria? Pudria? The package builder? Yeah. Currently, no. And Steve has tried it but which is not very practical and we are trying to improve this. Yeah, we followed Dev Summit yesterday. We have some new ideas and we'll be trying in the following weeks. So, any other? Oh, yeah. So, Pudria is actually able to provide hardware that's suitable for this and that's why we are running on Java and the build pipeline being an issue if you wanted to do native builds or something like that. Or... How do we build RISC 5 and the R64? Well, yeah, so you said you use KMU. Yes. And that's driven off of X86 but I think, is there ever a point where you'll be running the CI on native hardware or RISC 5 native? Let me repeat your question. Are you asking about is it possible to run architectures like RISC 5 or R64 on native machines, native hardware? No, it was more on the lines of... So, everything's running on X86 and then it's spawned, right? So, the spawner machine, you ever see that being like RMOR or RISC 5 in the future? Is that desirable or is that...? To run test on a physical machine? Yes. Yeah, it is possible and which... In fact, at least it is on our roadmap but this requires some more spatial hardware. We need to probably need JTAG also to automatically install a new image on that board and collect the console output and maybe another power controller to rebuild that board. Yeah. Okay. So, in that case, the Jenkins Java worker on the X86 platform... So, before the slave, you can just use a Sysage, you know, like we... You just spoke about the Sysage. Yeah, the build slave can still be a X86 but we just connect those development board to that machine and last slave just is queued, commenced through the connection of that board. That's it. So, since no question now, thanks everybody again. Yeah. Thank you.
So today, this talk is about continuous continuous integration CI this turn. And so it is just a software engineering practice of merging that we can know the conflicts and the know the bugs earlier as early as possible. So Wikipedia, And then we need some more work to make our build and sign history of our continuous integration. So in 2013, Craig started a group called Jenkins of the CI in the industry. So it's an open source automation server written in Java by CosK. And Jenkins can be extended by various plugins. It is the best part of it. So along with Jenkins admin pipeline, there are many object files or artifact files and the tests and do anything they want. So we introduced Artifact server, which will be to try to make each build jobs more organized. Then we create a test virtual machine image and machine image for people to try the latest run test. And where we run the test, we use of this, these scripts also extract the results the developer machine extract results. So how to track of the history of the test results. So you status of the test in each build. And for for now. First one is the disk files, which is then set up Nidia's version of FreeBSD. So we try they created a project called the Jenkins Drop code control system we like and use Jenkins Drop file. The most important part of this YAML file clean environment setup. So basically a job build.sh which is the mandatory. This is the main job directory. And we create a new entry in information to users. And some changes some are it found. So people can go through those files provide a warning or mostly warning resource for people to improve the documentation quality. So the quality of the documentation. We also do in the build we also collect warnings, compiler configuration to keep track of the... There are which cannot be published to the public. So we code management repository. And we also use set summary. Continuous integration does really help details. And we also want to build for the other downstream projects to make those ecosystem