From zero to first test in your own LAVA laboratory
Wieczorek, Paweł
Linaro Automated Validation Architecture (LAVA) is without a doubt one of the best currently available tools for managing QA board farms. It is proven to be quite a handy tool for both developers and tests automation engineers. Although it is provided together with extensive documentation, creating first own laboratory might be a challenging task. Does it have to be for every newcomer? During this talk Paweł will guide through the process of setting up own LAVA instance. He will also present how to manage its configuration and how to easily make deployments automated and reproducible.
 Hello everybody, my name is Paweł Wioczorek, I work for Samsung R&D Institute Poland and I'm currently a member of Tizen Release Engineering Team. Tizen is a GNU Linux distribution and in the common variant which I currently take care of, its main aim is to provide support for as many development boards as possible. That includes some popular ones like Raspberry Pi's and some less frequently used like Odroid's, Artix and so on. Today I would like to share with you a short introduction to what laboratory is. Since, as you might imagine, tasks such as validating, verifying, testing, operating systems on wide variety of development boards involves a lot of automation and in order to stay efficient we have to put as much of our work on automated systems. I will start with a short introduction to what Lava actually is, how you might benefit from it and what can it give you. Then I will go through the steps of actually setting up your own Lava laboratory. I would also like to share a few useful tools that might make your future work with Lava much easier. Next I will show you how you can get these results quickly and also I would like to share with you a few steps that you might go from having your first instance and what to do with it actually. I will conclude with a short summary, a few final thoughts and of course Q&A session. Let's start with what Lava actually is. Lava is an acronym which stands for linear automated validation architecture and it's an automated system for deploying operating systems on embedded devices but also not only just on the physical devices but also on the virtual ones like emulators or VMs. That's why we became interested in it in the first place, to make the whole process of deployment kernel, device tree blobs, root fs on the embedded devices with no manual interaction. As I mentioned, Lava lets us to test on both actual embedded devices and in the virtualized environment and once the operating system is put onto the device, a wide range of tests can be run on each of the devices and that includes boot tests even at the boot loader level or some higher level tests. Although it might be worth to note that some of the tests might require some additional hardware like the tests for power consumption or temperature depending on the load of the load on the devices. Let's ask ourselves when is it actually needed. In the most basic setup for embedded development like single ARM v7 board like Bigelbomb Black, it's not really hard to memorize all the steps for flashing the device, communicating with it, gathering results from the test runs but it has some downsides. For example, single developer can use it at the time. It's hard to share it without actual physical presence and passing it from one developer to another and it lacks running tests in parallel but that's not the only problem with this setup. What about if your application or your embedded GNU Linux distribution has to provide support for other target devices even ARM v7 based like the BBB? For example, Arctic 10 on the top left or Adroid Zoo 3 on the bottom right and even with all the procedures in place and all of that known by the developers, maybe your application or your distribution has to provide support for a completely new development board like Minobot Turbo which is Intel based and all of the operations that I mentioned earlier are done in a completely different way than the ARM based devices. Even if all of these details are shared among your team and it's no longer an issue, sooner or later the test results will have to be provided quicker, it will have to be done in parallel and it managing the whole board farm of yours will require too much effort from the developers. So maybe you could benefit from an abstraction layer put over your whole board farm. That's actually what Lava is. Lava allows to simplify even a complex ways of managing the devices. You don't have to worry about the specifics of each device like flashing, running, communicating, gathering results. That's no longer a concern of yours. You don't have to worry about device specific features as long as they were previously defined in your Lava configuration and you don't have to worry about managing multiple instances of your device types as well. The scheduling and dispatching of the tasks is done for you in the Lava environment. It provides a unified way of communicating with devices from developers point of view as long as all the devices were configured beforehand. Developers no longer have to put an effort into managing the devices. They look from their perspective equally the same. With Lava you also get to run all of your tests in parallel as long as they do not depend on each other and can be divided into smaller chunks. Also it stores and archives all of the results that you get over time so that you can review them or investigate more closely if you need to in the long run. Also you do not lose any of the benefits that you get from the direct interaction with your board. You can get to device directly by using either built-in solutions which are hacking sessions or some external tool like board overs here by free electrons. Since this tool gives you so much let's go through who actually uses Lava currently. First of all the team behind the tool itself, Linaro, currently uses it for testing both Linux and Android on development boards. Also the kernel CI project, people from kernel CI project who test the support of many development boards in the kernel perform boot tests within Lava laboratories. As for the whole distributions, currently both Debian and Automotive Grade Linux do their QA in Lava laboratories. Now that we have some rough image of how can we benefit from Lava, let's go through the setup steps. We will focus today on standalone instance even though Lava allows you to get your environment distributed. What I mean is that it doesn't matter whether the boards are in your European office or Asian office or any other country, it will be still the same from the Lava point of view. We'll also focus on the virtual devices only today and we will not get into the topic of writing tests for the Lava. Why is that? First of all, to make these first steps as straightforward as possible. Since it might be completely new work for you, it is really important to get familiar with all the basic concepts so that it will not get in your way and will help you instead of making things even more difficult. Even though the tests that you might already use can be reused in Lava, it might be wise to postpone the migration after the whole setup is already done. What are Lava's requirements? Fortunately, Lava is not too demanding. In the setup that we go through today, we will need only a supported Debian release which currently includes even the old stable Jesse. Currently, the Ubuntu support is frozen due to the old version of Django in default repositories. If you are interested in the details of this, you might find the proper mailing list thread linked on the page of this talk on program.froscon.de. Apart from the Debian-based platform, we will need also a few additional files. First of all, system image. It might be built all by yourselves or taken from the prebuilt images provided by Linaro on their main Lava instance on images.validation.linaro.org. For starters, it might be the quickest way to get up to speed. Does Lava support building your own images within the framework or do you have to build them outside and just supply? Currently, Lava needs to have these images already built. Some additional setup would be required. For example, as you mentioned, building them elsewhere and then supplying just to Lava. Next, you will also have to, apart from the operating system image, you will have to supply a health check job just to make sure that the board operates properly. These can also be obtained directly from Linaro on git.linaro.org under QA domain. You will find many exemplary health check jobs for various embedded devices. Also, device type template will be necessary, but those are also supplied by Linaro in all of the Lava installations. There are multiple templates already in place and the one that we will need for QMU is provided by default. The only file that you will have to prepare beforehand all by yourselves is this only three-line file which consists of the definition which template will be extended and two features that will be specified in order to avoid any conflicts between devices. The MAC address for the QMU device and for maximizing utilization of your resources, specifying the maximum memory of the device. Once you have all of this prepared, you might go directly to setting up. Thanks to the efforts from Lava packaging team, Debian repositories already provide Lava meta packages that we will use today. But beforehand, the database for the results, configuration and so on has to be prepared. It's also worth note that just for today we will use the meta package for Lava, but as your laboratories get more sophisticated and as your requirements grow, you might benefit from the fine grained packages to stay at the minimal possible setup of Lava and not to install any unnecessary parts. Once this is all in place, all you have to do to set up WebUI would be to enable two additional modules for Apache and replacing the default configuration with the one that is already supplied with your Lava server and of course, restarting your service in order to apply all the changes on your server. Also, do remember to set up the SuperUser4 for the whole configuration. As for actually adding devices to your laboratory, these three steps are everything that you need. First of all, adding the actual device type or letting the laboratory know what do you have in your laboratory. The first one has to be done only once per each device type. The next ones are adding instance of your device and actually specifying the details of your instance or in Lava terms, passing the device dictionary to your Lava laboratory. Once this is done, you might go to the CLI or WebUI and since it allows you to automate everything, I believe CLI will be the tool that you will use the most, but for just quick and dirty tests, the WebUI should be sufficient. There are also a few other tools that you might benefit from when you set up your own Lava laboratory. First of all, it might be beneficial to have your environment reproducible and from Lava point of view, it doesn't really matter what provider of configuration management you'll use, they all can perform equally well. But it is important to have your environment reproducible so that you will always be sure that your staging production or any other evaluation environment will always have the same setup. So choose your personal favorite in Tizen, we use Ansible Playbooks the most. Apart from that, I believe that your evaluation environment will probably be done in virtual machines. So depending on how much time do you have to spare, you might either use the Quicker solution, which is Vagrant Management Software for virtual machines, which allows you to bring up VMs almost instantly and provides a wide range of prebuilt boxes via its Atlas service. But do be careful, since not all the basic boxes are provided in every virtualization provider you might use. So take a closer look at what you will be pulling from the Atlas service. If you have some more time to spare and like to tinker a little bit, LibVirt will probably be the solution for you since it allows for the setup to be more adjusted to your needs, better adjusted to your needs and still comes with a few user friendly both CLI and GUI tools. Don't think that all of those steps that I already mentioned, that you will have to run them all by yourselves. Of course, it might be a good exercise to run them once more, but on the page of this talk, you will find an example of virtual machine configuration with Lava installed from the Meta package with QM device based on Vagrant LibVirt, virtualization provider and provisioned with Ansible Playbook. Let me just go real quick. What you will find in the tarball are actually basic requirements for this evaluation environment to use, which are currently only pip and Ansible installed on your host machine. Everything else will be brought up for you with playbooks from the tarball as well. How about now? Is it better? Let's stay with this font size. As for the configuration of your virtual machine, if you decide to modify it, do remember to allow the virtual machine to nest other virtual machines within it so that you will be able to add QM device as the KVM machine. All the steps that I already mentioned are written as Ansible roles. The database introduction, the setup steps for Lava and others are already done for you so that you will get up to speed in no time. Just to be sure, I will bring up the virtual machine. I have to admit that it is a little cheating. I brought it up just before the stock so that you won't have to wait for the download of all the packages. Once we have this configuration applied to our virtual machine with Lava evaluation environment, we can go to the first instance of your own Lava laboratory. For example, check out the devices that were added in the initial state, just the single QM device. As you can see, it already performed a few health checks. What is inside such a test job? It is just the basic template. It requires three main actions. The deployment instructions, boot instructions and actual test run. Deployment instructions involve what will be run on your device and how it will be run. As for boot, it defines what to expect from your operating systems and the actual tests are described in separate Git repositories available directly from Linaro. Now let's get back to the final nodes. What you might do with your own Lava laboratories from now? If you'd be interested in actually adding some physical devices, the documentation on that can be found on each of the Lava laboratories instances. The most recent documentation for that can be found in the main Lava instance on validation.linaro.org. The topic of actual writing tests is described in the chapter developing tests on the documentation site as well. You might also be interested in adding your laboratory to the Kernel CI project if you'd like to contribute some new boards or the ones that are already available. But you would like to get the results on a wider range of Kernel trees. I believe that everyone will benefit from new boards in Kernel CI laboratories. As for the articles that you might find interesting, Automotive Great Linux Distribution publishes the whole setup of their laboratory and test framework used in their QA activities. And testing initiative from civil infrastructure platform also provides many interesting materials on using laboratories, running tests on them, and managing the results. If you'd be more interested in watching a video or listening to it rather than reading an article, you might be interested in the talk from Bill Fletcher from last year's Linaro Connect, which goes into the detail on internals of Lava laboratories and the design of it. If you'd be interested in using direct access to development boards connected to your Lava laboratories, the talk from Antoine Tenard and Quentin Schultz from last year's ELCE, embedded Linux on Europe would be the one for you, or maybe you'd be interested in becoming familiar with the whole setup in the Linux distribution, then the Jan Simon Miller's talk would be the one for you. Of course, much more documentation is already available and this talk just touches the tip of the iceberg of what Lava actually is. And even though it might seem a little overwhelming at first, all the common issues and some difficulties that you might encounter will probably be all solved if you look into that documentation closely. If you have some specific issues, also Lava users mailing list or Linaro-Lava IRC channel on FreeNote will be probably the place you'll get your issues resolved at no time. And to sum it all up, thanks to the efforts from Lava packaging team, once you go through the whole documentation of setting up your first Lava laboratory and pick necessary instructions, you will get your first instance in no time if you meet all of the requirements. It allows you to unify the whole setup of your laboratory and you get the parallel execution of all your test cases with no responsibilities on the developers. The developers see the devices, the devices seem to be the same from the developers point of view. And even though the documentation for this project might be a little intimidating at first, the fact that you will get all of the answers from them says it's the right way to go. Also, there is absolutely no need to reinvent the board farm management software if the one that Linaro provides meets all of your requirements and allows you to make your workflow work more efficiently. And even though the way of automating all of these actions might seem a little costly at first, it will be beneficial in long run with each new test run that you will perform in your own Lava laboratories. And since you already have the starting points, you might get up to speed in no time. That will be all of what I've got prepared for you today. If you have any questions, I will be more than happy to answer them. What's the minimum requirements for the set of interfaces for a board to be used by Lava? Does it require Ethernet? The minimum requirement is depending on the way of deploying your images. If you have to perform only tests with benchmark possibilities of the board and you have some specific image, only a serial console just to communicate with the device will be sufficient. But if you'd like to deploy different images, then some other transport player would be more efficient. So some network connection might be helpful, but the absolute minimum is just the serial console. Thank you. Another question, sorry. So I can just have a board which just has serial and do everything with it, except it will be very slow. That's right. Okay, thank you. A question slightly more on the hardware side. If you want to have the ability of testing bootstrap procedures within your Fram, which might of course go wrong, a bootloader might not work, so the board doesn't come up. So one needs some mechanism to replace a bootloader, which normally means taking out storage device, reflashing SD card or whatever. Do we have experience with a hardware solution that allows remotely deploying boot images? Unfortunately, no. The requirement for Lava is to have a way of communicating with the operating system with the device already booted up. The other option than not booting from SD card would be to boot from NFS. Does Lava have plug-ins or the capability to kind of place an image on a NFS share and then kind of boot it up and then switch it around? Yes, there are multiple different ways of deploying images onto the devices and booting from NFS is one of the available ones. One of the recent changes, a pull request from AGL allows to boot the device directly from NBDs, from network block devices, if you'd be interested in such activity. What's the interface Lava uses to talk to tests? The interface for communicating with devices or... With the tests, like when it runs the tests, how it communicates the results, the output of the test. Does the test have to be written in a specific language or does it use some protocol to communicate with Lava to report the results? The tests are written in the Lava-compatible YAML configuration files, which have to comply with its pipeline schema. So the minimum requirement would be the three parts that I mentioned earlier. So the deployment stage, boot stage and the actual test run. But I can see that that might not be the answer for your question. Could you rephrase please? How do I make a test? Is it a separate executable or is it a library for Ruby or something? What is a test? The test for the Lava is... All right. So Lava deploys shell script onto your device. So if you can write your test as this set of steps that would be run otherwise manually by you through the terminal on your device, then you can also write it as the Lava YAML test template. Okay. Thank you. If there are some other questions that come up, feel free to drop me a line at my email address. Thanks for your attention and have a nice rest of your day. Thank you. Thank you.
 Hello everybody, my name is Paweł Wioczorek, I work for Samsung R&D Institute Poland and  I'm currently a member of Tizen Release Engineering Team.  Tizen is a GNU Linux distribution and in the common variant which I currently take care  of, its main aim is to provide support for as many development boards as possible.  That includes some popular ones like Raspberry Pi's and some less frequently used like  Odroid's, Artix and so on. Today I would like to share with you a short introduction to what  laboratory is. Since, as you might imagine, tasks such as validating, verifying, testing,  operating systems on wide variety of development boards involves a lot of automation and in  order to stay efficient we have to put as much of our work on automated systems.  I will start with a short introduction to what Lava actually is, how you might benefit from it  and what can it give you. Then I will go through the steps of actually setting up your own Lava  laboratory. I would also like to share a few useful tools that might make your future work with Lava  much easier. Next I will show you how you can get these results quickly and also I would like to  share with you a few steps that you might go from having your first instance and what to do with it  actually. I will conclude with a short summary, a few final thoughts and of course Q&A session.  Let's start with what Lava actually is. Lava is an acronym which stands for  linear automated validation architecture and it's an automated system for deploying operating  systems on embedded devices but also not only just on the physical devices but also on the virtual  ones like emulators or VMs. That's why we became interested in it in the first place,  to make the whole process of deployment kernel, device tree blobs, root fs on the embedded devices  with no manual interaction. As I mentioned, Lava lets us to test on both actual embedded devices  and in the virtualized environment and once the operating system is put onto the device,  a wide range of tests can be run on each of the devices and that includes boot tests even at  the boot loader level or some higher level tests. Although it might be worth to note that some of  the tests might require some additional hardware like the tests for power consumption or temperature  depending on the load of the load on the devices. Let's ask ourselves when is it actually needed.  In the most basic setup for embedded development like single ARM v7 board like Bigelbomb Black,  it's not really hard to memorize all the steps for flashing the device, communicating with it,  gathering results from the test runs but it has some downsides. For example,  single developer can use it at the time. It's hard to share it without actual physical presence  and passing it from one developer to another and it lacks running tests in parallel but that's not  the only problem with this setup. What about if your application or your embedded GNU Linux  distribution has to provide support for other target devices even ARM v7 based like the BBB?  For example, Arctic 10 on the top left or Adroid Zoo 3 on the bottom right and even with all the  procedures in place and all of that known by the developers, maybe your application or your  distribution has to provide support for a completely new development board like Minobot Turbo which  is Intel based and all of the operations that I mentioned earlier are done in a completely  different way than the ARM based devices. Even if all of these details are shared among your  team and it's no longer an issue, sooner or later the test results will have to be provided  quicker, it will have to be done in parallel and it managing the whole board farm of yours will  require too much effort from the developers. So maybe you could benefit from an abstraction layer  put over your whole board farm. That's actually what Lava is. Lava allows to simplify even a  complex ways of managing the devices. You don't have to worry about the specifics of each device  like flashing, running, communicating, gathering results. That's no longer a concern of yours.  You don't have to worry about device specific features as long as they were previously defined  in your Lava configuration and you don't have to worry about managing multiple instances of your  device types as well. The scheduling and dispatching of the tasks is done for you in the Lava  environment. It provides a unified way of communicating with devices from developers point  of view as long as all the devices were configured beforehand. Developers no longer have to put an  effort into managing the devices. They look from their perspective equally the same. With Lava  you also get to run all of your tests in parallel as long as they do not depend on each other and  can be divided into smaller chunks. Also it stores and archives all of the results that you get over  time so that you can review them or investigate more closely if you need to in the long run.  Also you do not lose any of the benefits that you get from the direct interaction with your board.  You can get to device directly by using either built-in solutions which are hacking sessions  or some external tool like board overs here by free electrons. Since this tool gives you so much  let's go through who actually uses Lava currently. First of all the team behind the tool itself,  Linaro, currently uses it for testing both Linux and Android on development boards. Also the kernel  CI project, people from kernel CI project who test the support of many development boards in the  kernel perform boot tests within Lava laboratories. As for the whole distributions, currently both  Debian and Automotive Grade Linux do their QA in Lava laboratories. Now that we have some rough  image of how can we benefit from Lava, let's go through the setup steps. We will focus today on  standalone instance even though Lava allows you to get your environment distributed. What I mean  is that it doesn't matter whether the boards are in your European office or Asian office or any other  country, it will be still the same from the Lava point of view. We'll also focus on the virtual  devices only today and we will not get into the topic of writing tests for the Lava. Why is that?  First of all, to make these first steps as straightforward as possible. Since it might be  completely new work for you, it is really important to get familiar with all the basic concepts so  that it will not get in your way and will help you instead of making things even more difficult.  Even though the tests that you might already use can be reused in Lava, it might be wise to postpone  the migration after the whole setup is already done. What are Lava's requirements? Fortunately,  Lava is not too demanding. In the setup that we go through today, we will need only a supported  Debian release which currently includes even the old stable Jesse. Currently, the Ubuntu support is  frozen due to the old version of Django in default repositories. If you are interested in the details  of this, you might find the proper mailing list thread linked on the page of this talk on  program.froscon.de. Apart from the Debian-based platform, we will need also a few additional  files. First of all, system image. It might be built all by yourselves or taken from the  prebuilt images provided by Linaro on their main Lava instance on images.validation.linaro.org.  For starters, it might be the quickest way to get up to speed. Does Lava support building  your own images within the framework or do you have to build them outside and just supply?  Currently, Lava needs to have these images already built. Some additional setup would be  required. For example, as you mentioned, building them elsewhere and then supplying just to Lava.  Next, you will also have to, apart from the operating system image, you will have to supply a  health check job just to make sure that the board operates properly. These can also be obtained  directly from Linaro on git.linaro.org under QA domain. You will find many exemplary health  check jobs for various embedded devices. Also, device type template will be necessary,  but those are also supplied by Linaro in all of the Lava installations. There are multiple  templates already in place and the one that we will need for QMU is provided by default.  The only file that you will have to prepare beforehand all by yourselves is this only three-line file  which consists of the definition which template will be extended and two features that will be  specified in order to avoid any conflicts between devices. The MAC address for the QMU device and  for maximizing utilization of your resources, specifying the maximum memory of the device.  Once you have all of this prepared, you might go directly to setting up. Thanks to the efforts  from Lava packaging team, Debian repositories already provide Lava meta packages that we will  use today. But beforehand, the database for the results, configuration and so on has to be prepared.  It's also worth note that just for today we will use the meta package for Lava,  but as your laboratories get more sophisticated and as your requirements grow,  you might benefit from the fine grained packages to stay at the minimal possible setup of Lava and  not to install any unnecessary parts. Once this is all in place, all you have to do to set up WebUI  would be to enable two additional modules for Apache and replacing the default configuration  with the one that is already supplied with your Lava server and of course, restarting your service  in order to apply all the changes on your server. Also, do remember to set up the SuperUser4  for the whole configuration. As for actually adding devices to your laboratory, these three steps are  everything that you need. First of all, adding the actual device type or letting the laboratory know  what do you have in your laboratory. The first one has to be done only once per each device type.  The next ones are adding instance of your device and actually specifying the details of your instance  or in Lava terms, passing the device dictionary to your Lava laboratory. Once this is done,  you might go to the CLI or WebUI and since it allows you to automate everything, I believe CLI  will be the tool that you will use the most, but for just quick and dirty tests, the WebUI should be sufficient.  There are also a few other tools that you might benefit from when you set up your own Lava laboratory.  First of all, it might be beneficial to have your environment reproducible and from Lava point of view,  it doesn't really matter what provider of configuration management you'll use, they all can perform equally well.  But it is important to have your environment reproducible so that you will always be sure that your staging production  or any other evaluation environment will always have the same setup. So choose your personal favorite  in Tizen, we use Ansible Playbooks the most. Apart from that, I believe that your evaluation environment  will probably be done in virtual machines. So depending on how much time do you have to spare,  you might either use the Quicker solution, which is Vagrant Management Software for virtual machines,  which allows you to bring up VMs almost instantly and provides a wide range of prebuilt boxes via its Atlas service.  But do be careful, since not all the basic boxes are provided in every virtualization provider you might use.  So take a closer look at what you will be pulling from the Atlas service.  If you have some more time to spare and like to tinker a little bit,  LibVirt will probably be the solution for you since it allows for the setup to be more adjusted to your needs,  better adjusted to your needs and still comes with a few user friendly both CLI and GUI tools.  Don't think that all of those steps that I already mentioned, that you will have to run them all by yourselves.  Of course, it might be a good exercise to run them once more, but on the page of this talk,  you will find an example of virtual machine configuration with Lava installed from the Meta package  with QM device based on Vagrant LibVirt, virtualization provider and provisioned with Ansible Playbook.  Let me just go real quick.  What you will find in the tarball are actually basic requirements for this evaluation environment to use,  which are currently only pip and Ansible installed on your host machine.  Everything else will be brought up for you with playbooks from the tarball as well.  How about now? Is it better? Let's stay with this font size.  As for the configuration of your virtual machine, if you decide to modify it, do remember to allow the virtual machine to nest other virtual machines within it  so that you will be able to add QM device as the KVM machine.  All the steps that I already mentioned are written as Ansible roles.  The database introduction, the setup steps for Lava and others are already done for you so that you will get up to speed in no time.  Just to be sure, I will bring up the virtual machine.  I have to admit that it is a little cheating.  I brought it up just before the stock so that you won't have to wait for the download of all the packages.  Once we have this configuration applied to our virtual machine with Lava evaluation environment,  we can go to the first instance of your own Lava laboratory.  For example, check out the devices that were added in the initial state, just the single QM device.  As you can see, it already performed a few health checks.  What is inside such a test job?  It is just the basic template.  It requires three main actions.  The deployment instructions, boot instructions and actual test run.  Deployment instructions involve what will be run on your device and how it will be run.  As for boot, it defines what to expect from your operating systems and the actual tests are described in separate Git repositories  available directly from Linaro.  Now let's get back to the final nodes.  What you might do with your own Lava laboratories from now?  If you'd be interested in actually adding some physical devices, the documentation on that can be found on each of the Lava laboratories instances.  The most recent documentation for that can be found in the main Lava instance on validation.linaro.org.  The topic of actual writing tests is described in the chapter developing tests on the documentation site as well.  You might also be interested in adding your laboratory to the Kernel CI project if you'd like to contribute some new boards or the ones that are already available.  But you would like to get the results on a wider range of Kernel trees.  I believe that everyone will benefit from new boards in Kernel CI laboratories.  As for the articles that you might find interesting, Automotive Great Linux Distribution publishes the whole setup of their laboratory  and test framework used in their QA activities.  And testing initiative from civil infrastructure platform also provides many interesting materials on using laboratories, running tests on them, and managing the results.  If you'd be more interested in watching a video or listening to it rather than reading an article, you might be interested in the talk from Bill Fletcher from last year's Linaro Connect,  which goes into the detail on internals of Lava laboratories and the design of it.  If you'd be interested in using direct access to development boards connected to your Lava laboratories, the talk from Antoine Tenard and Quentin Schultz from last year's ELCE,  embedded Linux on Europe would be the one for you, or maybe you'd be interested in becoming familiar with the whole setup in the Linux distribution,  then the Jan Simon Miller's talk would be the one for you.  Of course, much more documentation is already available and this talk just touches the tip of the iceberg of what Lava actually is.  And even though it might seem a little overwhelming at first, all the common issues and some difficulties that you might encounter will probably be all solved if you look into that documentation closely.  If you have some specific issues, also Lava users mailing list or Linaro-Lava IRC channel on FreeNote will be probably the place you'll get your issues resolved at no time.  And to sum it all up, thanks to the efforts from Lava packaging team, once you go through the whole documentation of setting up your first Lava laboratory and pick necessary instructions,  you will get your first instance in no time if you meet all of the requirements.  It allows you to unify the whole setup of your laboratory and you get the parallel execution of all your test cases with no responsibilities on the developers.  The developers see the devices, the devices seem to be the same from the developers point of view.  And even though the documentation for this project might be a little intimidating at first, the fact that you will get all of the answers from them says it's the right way to go.  Also, there is absolutely no need to reinvent the board farm management software if the one that Linaro provides meets all of your requirements and allows you to make your workflow work more efficiently.  And even though the way of automating all of these actions might seem a little costly at first, it will be beneficial in long run with each new test run that you will perform in your own Lava laboratories.  And since you already have the starting points, you might get up to speed in no time.  That will be all of what I've got prepared for you today. If you have any questions, I will be more than happy to answer them.  What's the minimum requirements for the set of interfaces for a board to be used by Lava? Does it require Ethernet?  The minimum requirement is depending on the way of deploying your images. If you have to perform only tests with benchmark possibilities of the board and you have some specific image, only a serial console just to communicate with the device will be sufficient.  But if you'd like to deploy different images, then some other transport player would be more efficient.  So some network connection might be helpful, but the absolute minimum is just the serial console.  Thank you. Another question, sorry. So I can just have a board which just has serial and do everything with it, except it will be very slow.  That's right. Okay, thank you.  A question slightly more on the hardware side. If you want to have the ability of testing bootstrap procedures within your Fram, which might of course go wrong, a bootloader might not work, so the board doesn't come up.  So one needs some mechanism to replace a bootloader, which normally means taking out storage device, reflashing SD card or whatever. Do we have experience with a hardware solution that allows remotely deploying boot images?  Unfortunately, no. The requirement for Lava is to have a way of communicating with the operating system with the device already booted up.  The other option than not booting from SD card would be to boot from NFS. Does Lava have plug-ins or the capability to kind of place an image on a NFS share and then kind of boot it up and then switch it around?  Yes, there are multiple different ways of deploying images onto the devices and booting from NFS is one of the available ones.  One of the recent changes, a pull request from AGL allows to boot the device directly from NBDs, from network block devices, if you'd be interested in such activity.  What's the interface Lava uses to talk to tests?  The interface for communicating with devices or...  With the tests, like when it runs the tests, how it communicates the results, the output of the test. Does the test have to be written in a specific language or does it use some protocol to communicate with Lava to report the results?  The tests are written in the Lava-compatible YAML configuration files, which have to comply with its pipeline schema.  So the minimum requirement would be the three parts that I mentioned earlier. So the deployment stage, boot stage and the actual test run.  But I can see that that might not be the answer for your question. Could you rephrase please?  How do I make a test? Is it a separate executable or is it a library for Ruby or something? What is a test?  The test for the Lava is... All right. So Lava deploys shell script onto your device. So if you can write your test as this set of steps that would be run otherwise manually by you through the terminal on your device,  then you can also write it as the Lava YAML test template.  Okay. Thank you.  If there are some other questions that come up, feel free to drop me a line at my email address. Thanks for your attention and have a nice rest of your day. Thank you.  Thank you.
froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_0_59.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_60_2207.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_2208_2315.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_2316_2387.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_2388_2459.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_2460_3623.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_3624_3707.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_3708_6371.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_6372_7871.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_7872_9683.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_9684_10523.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_10524_24779.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_24780_27071.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_27072_29039.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_29040_29375.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_29376_32615.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_32616_34775.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_34776_35123.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_35124_35447.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_35448_35531.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_35532_35675.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_35676_36779.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_36780_36983.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_36984_37031.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_37032_37295.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_37296_37487.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_37488_37751.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_37752_38567.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_38568_38615.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_38616_38915.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_38916_38987.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_38988_39047.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_39048_39851.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_39852_40343.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_40344_41099.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_41100_41147.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_41148_47207.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_47208_50879.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_50880_51047.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_51048_55043.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_55044_55247.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_55248_55391.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_55392_55727.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_55728_56123.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_56124_59195.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_59196_59447.png froscon2017-2048-from_zero_to_first_test_in_your_own_lava_laboratory_59448_59874.png
Hello everybody, my name is Paweł Wioczorek, I work for Samsung R&D Institute Poland and I'm currently a member of Tizen Release Engineering Team. Tizen is a GNU Linux distribution and in the common variant which I currently take care of, its main aim is to provide support for as many development boards as possible. That includes some popular ones like Raspberry Pi's and some less frequently used like Odroid's, Artix and so on. Today I would like to share with you a short introduction to what laboratory is. Since, as you might imagine, tasks such as validating, verifying, testing, operating systems on wide variety of development boards involves a lot of automation and in order to stay efficient we have to put as much of our work on automated systems. I will start with a short introduction to what Lava actually is, how you might benefit from it and what can it give you. Then I will go through the steps of actually setting up your own Lava laboratory. I would also like to share a few useful tools that might make your future work with Lava much easier. Next I will show you how you can get these results quickly and also I would like to share with you a few steps that you might go from having your first instance and what to do with it actually. I will conclude with a short summary, a few final thoughts and of course Q&A session. Let's start with what Lava actually is. Lava is an acronym which stands for linear automated validation architecture and it's an automated system for deploying operating systems on embedded devices but also not only just on the physical devices but also on the virtual ones like emulators or VMs. That's why we became interested in it in the first place, to make the whole process of deployment kernel, device tree blobs, root fs on the embedded devices with no manual interaction. As I mentioned, Lava lets us to test on both actual embedded devices and in the virtualized environment and once the operating system is put onto the device, a wide range of tests can be run on each of the devices and that includes boot tests even at the boot loader level or some higher level tests. Although it might be worth to note that some of the tests might require some additional hardware like the tests for power consumption or temperature depending on the load of the load on the devices. Let's ask ourselves when is it actually needed. In the most basic setup for embedded development like single ARM v7 board like Bigelbomb Black, it's not really hard to memorize all the steps for flashing the device, communicating with it, gathering results from the test runs but it has some downsides. For example, single developer can use it at the time. It's hard to share it without actual physical presence and passing it from one developer to another and it lacks running tests in parallel but that's not the only problem with this setup. What about if your application or your embedded GNU Linux distribution has to provide support for other target devices even ARM v7 based like the BBB? For example, Arctic 10 on the top left or Adroid Zoo 3 on the bottom right and even with all the procedures in place and all of that known by the developers, maybe your application or your distribution has to provide support for a completely new development board like Minobot Turbo which is Intel based and all of the operations that I mentioned earlier are done in a completely different way than the ARM based devices. Even if all of these details are shared among your team and it's no longer an issue, sooner or later the test results will have to be provided quicker, it will have to be done in parallel and it managing the whole board farm of yours will require too much effort from the developers. So maybe you could benefit from an abstraction layer put over your whole board farm. That's actually what Lava is. Lava allows to simplify even a complex ways of managing the devices. You don't have to worry about the specifics of each device like flashing, running, communicating, gathering results. That's no longer a concern of yours. You don't have to worry about device specific features as long as they were previously defined in your Lava configuration and you don't have to worry about managing multiple instances of your device types as well. The scheduling and dispatching of the tasks is done for you in the Lava environment. It provides a unified way of communicating with devices from developers point of view as long as all the devices were configured beforehand. Developers no longer have to put an effort into managing the devices. They look from their perspective equally the same. With Lava you also get to run all of your tests in parallel as long as they do not depend on each other and can be divided into smaller chunks. Also it stores and archives all of the results that you get over time so that you can review them or investigate more closely if you need to in the long run. Also you do not lose any of the benefits that you get from the direct interaction with your board. You can get to device directly by using either built-in solutions which are hacking sessions or some external tool like board overs here by free electrons. Since this tool gives you so much let's go through who actually uses Lava currently. First of all the team behind the tool itself, Linaro, currently uses it for testing both Linux and Android on development boards. Also the kernel CI project, people from kernel CI project who test the support of many development boards in the kernel perform boot tests within Lava laboratories. As for the whole distributions, currently both Debian and Automotive Grade Linux do their QA in Lava laboratories. Now that we have some rough image of how can we benefit from Lava, let's go through the setup steps. We will focus today on standalone instance even though Lava allows you to get your environment distributed. What I mean is that it doesn't matter whether the boards are in your European office or Asian office or any other country, it will be still the same from the Lava point of view. We'll also focus on the virtual devices only today and we will not get into the topic of writing tests for the Lava. Why is that? First of all, to make these first steps as straightforward as possible. Since it might be completely new work for you, it is really important to get familiar with all the basic concepts so that it will not get in your way and will help you instead of making things even more difficult. Even though the tests that you might already use can be reused in Lava, it might be wise to postpone the migration after the whole setup is already done. What are Lava's requirements? Fortunately, Lava is not too demanding. In the setup that we go through today, we will need only a supported Debian release which currently includes even the old stable Jesse. Currently, the Ubuntu support is frozen due to the old version of Django in default repositories. If you are interested in the details of this, you might find the proper mailing list thread linked on the page of this talk on program.froscon.de. Apart from the Debian-based platform, we will need also a few additional files. First of all, system image. It might be built all by yourselves or taken from the prebuilt images provided by Linaro on their main Lava instance on images.validation.linaro.org. For starters, it might be the quickest way to get up to speed. Does Lava support building your own images within the framework or do you have to build them outside and just supply? Currently, Lava needs to have these images already built. Some additional setup would be required. For example, as you mentioned, building them elsewhere and then supplying just to Lava. Next, you will also have to, apart from the operating system image, you will have to supply a health check job just to make sure that the board operates properly. These can also be obtained directly from Linaro on git.linaro.org under QA domain. You will find many exemplary health check jobs for various embedded devices. Also, device type template will be necessary, but those are also supplied by Linaro in all of the Lava installations. There are multiple templates already in place and the one that we will need for QMU is provided by default. The only file that you will have to prepare beforehand all by yourselves is this only three-line file which consists of the definition which template will be extended and two features that will be specified in order to avoid any conflicts between devices. The MAC address for the QMU device and for maximizing utilization of your resources, specifying the maximum memory of the device. Once you have all of this prepared, you might go directly to setting up. Thanks to the efforts from Lava packaging team, Debian repositories already provide Lava meta packages that we will use today. But beforehand, the database for the results, configuration and so on has to be prepared. It's also worth note that just for today we will use the meta package for Lava, but as your laboratories get more sophisticated and as your requirements grow, you might benefit from the fine grained packages to stay at the minimal possible setup of Lava and not to install any unnecessary parts. Once this is all in place, all you have to do to set up WebUI would be to enable two additional modules for Apache and replacing the default configuration with the one that is already supplied with your Lava server and of course, restarting your service in order to apply all the changes on your server. Also, do remember to set up the SuperUser4 for the whole configuration. As for actually adding devices to your laboratory, these three steps are everything that you need. First of all, adding the actual device type or letting the laboratory know what do you have in your laboratory. The first one has to be done only once per each device type. The next ones are adding instance of your device and actually specifying the details of your instance or in Lava terms, passing the device dictionary to your Lava laboratory. Once this is done, you might go to the CLI or WebUI and since it allows you to automate everything, I believe CLI will be the tool that you will use the most, but for just quick and dirty tests, the WebUI should be sufficient. There are also a few other tools that you might benefit from when you set up your own Lava laboratory. First of all, it might be beneficial to have your environment reproducible and from Lava point of view, it doesn't really matter what provider of configuration management you'll use, they all can perform equally well. But it is important to have your environment reproducible so that you will always be sure that your staging production or any other evaluation environment will always have the same setup. So choose your personal favorite in Tizen, we use Ansible Playbooks the most. Apart from that, I believe that your evaluation environment will probably be done in virtual machines. So depending on how much time do you have to spare, you might either use the Quicker solution, which is Vagrant Management Software for virtual machines, which allows you to bring up VMs almost instantly and provides a wide range of prebuilt boxes via its Atlas service. But do be careful, since not all the basic boxes are provided in every virtualization provider you might use. So take a closer look at what you will be pulling from the Atlas service. If you have some more time to spare and like to tinker a little bit, LibVirt will probably be the solution for you since it allows for the setup to be more adjusted to your needs, better adjusted to your needs and still comes with a few user friendly both CLI and GUI tools. Don't think that all of those steps that I already mentioned, that you will have to run them all by yourselves. Of course, it might be a good exercise to run them once more, but on the page of this talk, you will find an example of virtual machine configuration with Lava installed from the Meta package with QM device based on Vagrant LibVirt, virtualization provider and provisioned with Ansible Playbook. Let me just go real quick. What you will find in the tarball are actually basic requirements for this evaluation environment to use, which are currently only pip and Ansible installed on your host machine. Everything else will be brought up for you with playbooks from the tarball as well. How about now? Is it better? Let's stay with this font size. As for the configuration of your virtual machine, if you decide to modify it, do remember to allow the virtual machine to nest other virtual machines within it so that you will be able to add QM device as the KVM machine. All the steps that I already mentioned are written as Ansible roles. The database introduction, the setup steps for Lava and others are already done for you so that you will get up to speed in no time. Just to be sure, I will bring up the virtual machine. I have to admit that it is a little cheating. I brought it up just before the stock so that you won't have to wait for the download of all the packages. Once we have this configuration applied to our virtual machine with Lava evaluation environment, we can go to the first instance of your own Lava laboratory. For example, check out the devices that were added in the initial state, just the single QM device. As you can see, it already performed a few health checks. What is inside such a test job? It is just the basic template. It requires three main actions. The deployment instructions, boot instructions and actual test run. Deployment instructions involve what will be run on your device and how it will be run. As for boot, it defines what to expect from your operating systems and the actual tests are described in separate Git repositories available directly from Linaro. Now let's get back to the final nodes. What you might do with your own Lava laboratories from now? If you'd be interested in actually adding some physical devices, the documentation on that can be found on each of the Lava laboratories instances. The most recent documentation for that can be found in the main Lava instance on validation.linaro.org. The topic of actual writing tests is described in the chapter developing tests on the documentation site as well. You might also be interested in adding your laboratory to the Kernel CI project if you'd like to contribute some new boards or the ones that are already available. But you would like to get the results on a wider range of Kernel trees. I believe that everyone will benefit from new boards in Kernel CI laboratories. As for the articles that you might find interesting, Automotive Great Linux Distribution publishes the whole setup of their laboratory and test framework used in their QA activities. And testing initiative from civil infrastructure platform also provides many interesting materials on using laboratories, running tests on them, and managing the results. If you'd be more interested in watching a video or listening to it rather than reading an article, you might be interested in the talk from Bill Fletcher from last year's Linaro Connect, which goes into the detail on internals of Lava laboratories and the design of it. If you'd be interested in using direct access to development boards connected to your Lava laboratories, the talk from Antoine Tenard and Quentin Schultz from last year's ELCE, embedded Linux on Europe would be the one for you, or maybe you'd be interested in becoming familiar with the whole setup in the Linux distribution, then the Jan Simon Miller's talk would be the one for you. Of course, much more documentation is already available and this talk just touches the tip of the iceberg of what Lava actually is. And even though it might seem a little overwhelming at first, all the common issues and some difficulties that you might encounter will probably be all solved if you look into that documentation closely. If you have some specific issues, also Lava users mailing list or Linaro-Lava IRC channel on FreeNote will be probably the place you'll get your issues resolved at no time. And to sum it all up, thanks to the efforts from Lava packaging team, once you go through the whole documentation of setting up your first Lava laboratory and pick necessary instructions, you will get your first instance in no time if you meet all of the requirements. It allows you to unify the whole setup of your laboratory and you get the parallel execution of all your test cases with no responsibilities on the developers. The developers see the devices, the devices seem to be the same from the developers point of view. And even though the documentation for this project might be a little intimidating at first, the fact that you will get all of the answers from them says it's the right way to go. Also, there is absolutely no need to reinvent the board farm management software if the one that Linaro provides meets all of your requirements and allows you to make your workflow work more efficiently. And even though the way of automating all of these actions might seem a little costly at first, it will be beneficial in long run with each new test run that you will perform in your own Lava laboratories. And since you already have the starting points, you might get up to speed in no time. That will be all of what I've got prepared for you today. If you have any questions, I will be more than happy to answer them. What's the minimum requirements for the set of interfaces for a board to be used by Lava? Does it require Ethernet? The minimum requirement is depending on the way of deploying your images. If you have to perform only tests with benchmark possibilities of the board and you have some specific image, only a serial console just to communicate with the device will be sufficient. But if you'd like to deploy different images, then some other transport player would be more efficient. So some network connection might be helpful, but the absolute minimum is just the serial console. Thank you. Another question, sorry. So I can just have a board which just has serial and do everything with it, except it will be very slow. That's right. Okay, thank you. A question slightly more on the hardware side. If you want to have the ability of testing bootstrap procedures within your Fram, which might of course go wrong, a bootloader might not work, so the board doesn't come up. So one needs some mechanism to replace a bootloader, which normally means taking out storage device, reflashing SD card or whatever. Do we have experience with a hardware solution that allows remotely deploying boot images? Unfortunately, no. The requirement for Lava is to have a way of communicating with the operating system with the device already booted up. The other option than not booting from SD card would be to boot from NFS. Does Lava have plug-ins or the capability to kind of place an image on a NFS share and then kind of boot it up and then switch it around? Yes, there are multiple different ways of deploying images onto the devices and booting from NFS is one of the available ones. One of the recent changes, a pull request from AGL allows to boot the device directly from NBDs, from network block devices, if you'd be interested in such activity. What's the interface Lava uses to talk to tests? The interface for communicating with devices or... With the tests, like when it runs the tests, how it communicates the results, the output of the test. Does the test have to be written in a specific language or does it use some protocol to communicate with Lava to report the results? The tests are written in the Lava-compatible YAML configuration files, which have to comply with its pipeline schema. So the minimum requirement would be the three parts that I mentioned earlier. So the deployment stage, boot stage and the actual test run. But I can see that that might not be the answer for your question. Could you rephrase please? How do I make a test? Is it a separate executable or is it a library for Ruby or something? What is a test? The test for the Lava is... All right. So Lava deploys shell script onto your device. So if you can write your test as this set of steps that would be run otherwise manually by you through the terminal on your device, then you can also write it as the Lava YAML test template. Okay. Thank you. If there are some other questions that come up, feel free to drop me a line at my email address. Thanks for your attention and have a nice rest of your day. Thank you. Thank you.
useful tools that might make your future work automated validation architecture and it's an whole process of deployment kernel, device tree to be provided quicker, it will have to be done distributions, currently both Debian and supplying just to Lava. Next, you will also have configuration with the one that is already first instance of your own Lava laboratory. For go through the whole documentation of setting up documentation for this project might be a little Lava? Does it require Ethernet? The minimum with benchmark possibilities of the board and you the interface Lava uses to talk to tests? The