Dockerized pytests
Rafuls, Gonzalo
Dockerized pytests When working with pytest and custom plugins things can get eery when trying to mimic your Jenkins environment for debugging your tests locally, specially across your team and their workspaces. In this talk will go through the challenges faced to containerize a wild pytest environment. Topics: Running Pytest on Jenkins jobs Pytest custom plugins Challenges for locally debugging/replicating a Jenkins job/pytest run Dockerizing pytest and it's dependencie
 Thank you all guys for coming. I wasn't expecting this many people, but that's a good sign, I guess. First let me start a little bit by introducing myself. My name is Gonzalo Raffuls and I do quality engineering for Red Hat in Czech Republic for its virtualization solution which is over on the AppStream. So during this talk I will try to go fast and make it light, it's been a long week. To begin with I'm going to be talking a little bit about, I'm going to actually show you a simple test scenario for running PyTest on Jenkins and we're going to have a little bit of a high overview on PyTest custom plugins and how PyTest loads custom plugins. Then we will look a little bit at the challenges faced at the time of trying to replicate this Jenkins job on your local environment with Docker. We will look also at Docker as a solution for doing this and then if we have enough time I'll make a small live demo if it doesn't fail miserably. Now I'm not going to be talking, I'm not going to be, I don't want to intend to sweet talk you and to start using containers, you should be aware of the security implications that comes with containerizing your applications. We're not going to talk about containerizing Jenkins, that's already been done. It's ideal if you already have some high overview on Jenkins, PyTest and Docker and also most people would think that this is about the Docker Py library which is for managing your containers through Python code but that's not the case in this talk. Now, so why did I decide to talk about this and how did we came up with this solution for containerizing Jenkins? Now when I first came to Red Hat I was given the task of replicating my Jenkins job, our testing framework on my local environment and for doing so I was given a six-page document that it actually took me approximately two weeks to complete and then when I realized how big of a fool's errand it was to try and do so plus I also realized that most of the people on my team was having issues when trying to replicate their environments as well but at the same time those environments that were successfully working at some point with the constant evolution of our testing framework, those setups were easily lost. So there had to be a way to keep up with this and have it centralized and make sure that no one had to waste two weeks of their time to set it up. Now at the time we were not using virtual environments therefore you had to install all the requirements from the testing framework on your local machine and that could potentially cause some dependency issues on the libraries that you already had installed on your machine. Now also like the biggest benefit of this was to be able to put a Docker image on an internal registry that you can run on your organization so that whenever a newcomer started working just by pulling the image would be able to start debugging straight away. Now how many of you have been using PyTest on Jenkins already? Okay, that's a good number. So there's a couple of things that you need to take into consideration at the time of trying to replicate this Jenkins environment or this job run. So first thing, if the job that you're trying to replicate is PyTest you should look at the PyTest options on your job, on the configuration of your job and here you can have all sorts of things and these parameters can be used by Jenkins in many other places during the job execution. In the case that I will show you in this small sample scenario, we only have a parameter that will be used on one of the build steps that I will show you further on. Now you might also have distributed tests and you could also have parameters for determining to which hosts you want or on which hosts you want the job to run. So that's one thing to take into consideration. And if the job itself has any Git integration and if you're actually this is the most common practice so you probably do. So it's important to check that all those repositories are going to be used for our local environment. So this is really important and we will actually link all those repositories to volumes on our containers. Now finally I would look at the build steps on the configuration of the Jenkins job and it will depend on the complexity of your testing framework but you might have in this case I just have one build step where it's actually a dynamic bar script that takes one of the parameters that I defined before for choosing which tests to run. And you might have additional build steps that could, for example, in our case, generate configuration files that are used by custom plugins and you need to make sure that all those steps are covered before you decide to run PyTest on your container. Now with PyTest virtually any Python module can be registered as a plugin. So how PyTest does the plugin discovery, how it checks all the plugins that are on your testing framework, it first starts by loading all the built-in plugins that were installed with PyTest. Secondly it looks for, it actually loads all the plugins that are registered through setup tools entry point. I will show you a small sample on the next slide where I am actually passing one plugin that is outside our testing framework through the setup tools. Third thing that it does is pre-scans the command line for the minus p name option and it loads the specified plugin before doing the actual command line parsing of PyTest. In this case you can also choose to deactivate already registered plugins by prefixing the minus p option. Actually I didn't expect this to show like that but it's minus p, no semicolon and the name of the plugin. So that's how you avoid loading plugins that perhaps are causing issues on your testing framework. Next thing that PyTest loads, it looks for all the comftest.py files as inferred by the command line invocation and if you don't have any test paths specified, it uses the current directory as a path. Now you need to note that PyTest does not find comftest.py files in deeper nested subdirectories at tool start-up. So it is usually a good idea to keep your comftest.py files in the top level of your root project. Now next it does also load recursively all the plugins specified by the PyTest underscore plugins variable that you can specify inside your comftest. So that is actually the last place where PyTest looks for custom plugins. So now if you want to make your plugin externally available, you may define what is so-called entry point for your distribution so that PyTest finds your plugin module. Entry points are a feature that is provided by setup tools and PyTest looks specifically for the PyTest 11, I cannot see now here, it looks for this entry point and it actually looks for this entry point and it loads all the plugins that you have defined under this entry point. So now when I first started to try to containerize this environment, I started by creating a base image with Fedora and a couple of instructions that I already knew that had to be there. For example, one of the first things that I had to do was at the time of linking my local repositories to volumes inside my container, I realized that I had to include all these repositories into my Python path. So I managed to do that by including the M instruction on the Docker file for creating the image. And later on, I started trying to install all the requirements that were defined on our testing framework requirements.txt file inside the root directory of the testing framework. And it was then when I realized that some of the requirements that I was trying to install with PIP had header files from RPMs that were missing or were not included on any place on my testing framework. Because all these dependencies were either already included in the operating system that was being used for running these tests or being provisioned by Foreman or other services that were invisible to my testing framework. Also how I managed to see which plugins were loaded by PyTest at every point while I was trying to containerize these was by using the trace config option from PyTest which actually shows you all the plugins that are being recognized by PyTest at the time. Now as you can see there, I have this run statement that I included on my Docker file. Now it is recommended that you minimize the number of run commands because in that way you, it makes for less overhead at the time of actually running your container. So now when you are leaking a volume on your container to local repo directory on your machine, if you have the SE Linux policy enforced, then the container won't be able to write to this directory. And in the case that it might be possible that your tests are generating log files that are saved on that same repository. And if you have SE Linux enforced, the container will fail with permission denied or an IBC message in the host syslog. So to avoid that, you need to change the context of the directory, in this case the repository directory to that tag in particular, sv sandbox file t. Now this has become much more easier since Docker 1.7 because as you can see on the second piece of code here, I am preparing the set at the end. And since Docker 1.7, this statement automatically changes the context of the directory that you are trying to link to your container. Therefore, the container will be able to write to that repository. Now further on, at the very end of my Docker file, I define these two instructions, the entry point and the cmd. Now the entry point, it virtually makes your container executable. How so? Because when you run your container image, the first command that the container will run will be the entry point. And the cmd, it's any additional options that you would like to run with your entry point. But why is it separated like this? It's because the cmd is overwriteable. You can overwrite that by adding options to your run statement for when you want to run your container. And therefore, it makes it more flexible. Now at the time of debugging your PyTests, you might want to, I'm not sure, I use IPDB, for example, but unless you use PDB or PyTests on PDB, you won't be able to debug because PyTests is capturing the standard output. So you need to run PyTests with the minus S option so that you are able to use any other debuggers apart from PDB or PyTests PDB. So I went through that real fast. Now I will show you a little bit of a live demo and a small scenario that I made with Jenkins. You can find the testing framework that we are going to be using on this live demo on this repository. And the second repository contains the Docker files and the Docker compose file that I am using for instantiating these Jenkins environment. So I'm not sure if I'm able to see here. So this is Jenkins instance that I have running on a container, of course, in my local machine. And this job in particular is a parameterized job that takes only one parameter and based on the input that we put there, it will run one test or the other as we saw on the bash script before. So I will run a bad test and a good test. One should fail, the other one shouldn't. Most likely both will fail. So we got a failing test here. It's success. Let's say I want to debug this failing test that is running on my Jenkins instance. Now I already created a container image that will replicate my Jenkins instance and a container that I can use for debugging my Python locally. Now I have some scripts here. So if I run this script for running this bad test, I can show you what does this file contain. It's basically a run instruction for Docker to run with the FC24 container that I created with this Docker file. And basically I am telling straight away to PyTest which test to run. So now in theory, if I... This is the testing framework. So in theory, if I edit my bad test to include this breakpoint, then I should be able now to run my container and I should be able to hit that breakpoint. And yeah, there you see. So you can... Why don't you run on one? Because that actually doesn't... It actually hits the breakpoint when an exception is raised. So it actually stops the execution whenever a trace bug is raised. Now you might have no failing tests that you will still want to debug. So... Okay. So let me show you... I got another script here as well. And let me show you... I have defined on my Conf test, I have defined a small plugin, a small custom plugin that is in the form of a mark. And I am using that on my good file. No, sorry. So basically this mark is for including an option name on the PyTest run that will look for any tests that are tagged with whatever I introduce on my PyTest run. So if I go here... Now... Yeah. So if I run the good file now, this file in fact has... As you can see, I am passing the minus E option and the good tab that was defined on my marker for that test in particular. So if I try to run this container and I change the... I change the label to whatever, it will actually skip the test because it's not the label for good that we had defined on that test. Now additionally, we can see with the trace config, this is also just running PyTest with the trace config. On the trace config you will see all the planes that have been registered by PyTest as well as the comftests that have been loaded by... So if I do this, you can see... In this case, it's showing me that it has loaded the comftest from the root directory which is on the container itself because I am routing my volumes... I am linking my volumes to the main root folder on my container. So that's pretty much it. Now... Yeah. So, yeah, that's pretty much it. Now if you have any questions, I'm afraid we don't have any more time. Do we? Okay. Yeah, sure. I'm sorry, man. For this container in particular? Yeah. So this is the one for the Jenkins instance that it's actually picking up the base image from Docker Hub, this Jenkins latest. And I include some other instructions for what I needed at the time. Then you get the Jenkins slave Docker file which is just the Java container, base container. And there I am downloading the Java agent from Jenkins for executing the file then. So you can find this on GitHub. There is... You can find the Docker files there and... I should be... The Docker files where you had entry points and stuff, where are they? Ah, well, for this talk in particular, I didn't use any entry points or CMD, but it's like that. It's just like you can only use CMD once and the entry point should be your default command to be executed with the Docker. Hello. Thank you for the presentation. It was a very nice setup. Sorry, I cannot hear you. Thank you for the presentation. That was a nice setup. I have two questions. First of all, this looks like one of the benefits of this might be able to run multiple tests from Jenkins at the same time. Is this something that you've tried and it works properly with the setup? Yeah, you can run any test that you have on your testing framework if you already have it set up like that on your Jenkins instance. So this Jenkins instance was just queuing them because that's how Jenkins was set up? This Jenkins instance was just to show the two different places where the Jenkins job that is running the PyTest and my local PyTest run that will be able to debug the test. Okay, cool. My next question, can you tell me what that shell you had, it showed whether the git was dirty or clean? It looked very nice. I liked it. This is all my set SH with some tweaks. Okay, thank you. Hello. Thank you for the talk. It's very important. My question is when it comes to deploying your application, I can understand the benefits of using Docker. There are a lot of advantages. Like encapsulating all of the businesses inside your containers so you can ship it as is or make your application cloud native ready, something like that. But talking about testing only, what is the real advantage of using the containerized solution with respect to other available solutions that in my opinion might be more lightweight solutions like virtual environment? So have you considered this other solution? What is your opinion? As I said before, given the complexity of our testing framework, this was the best solution at the time. And as I said also, I don't want to preach or sweet talk you into using Docker or containers for replicating your Jenkins instance. But this is actually saving us a lot of time for newcomers and also it's maintaining some sort of uniformity on our testing framework. Do we have any more questions? No. Thank you very much Gonzalo. Thank you Gonzalo once more.
 Thank you all guys for coming.  I wasn't expecting this many people, but that's a good sign, I guess.  First let me start a little bit by introducing myself.  My name is Gonzalo Raffuls and I do quality engineering for Red Hat in Czech Republic  for its virtualization solution which is over on the AppStream.  So during this talk I will try to go fast and make it light, it's been a long week.  To begin with I'm going to be talking a little bit about, I'm going to actually show you  a simple test scenario for running PyTest on Jenkins and we're going to have a little  bit of a high overview on PyTest custom plugins and how PyTest loads custom plugins.  Then we will look a little bit at the challenges faced at the time of trying to replicate this  Jenkins job on your local environment with Docker.  We will look also at Docker as a solution for doing this and then if we have enough  time I'll make a small live demo if it doesn't fail miserably.  Now I'm not going to be talking, I'm not going to be, I don't want to intend to sweet  talk you and to start using containers, you should be aware of the security implications  that comes with containerizing your applications.  We're not going to talk about containerizing Jenkins, that's already been done.  It's ideal if you already have some high overview on Jenkins, PyTest and Docker and also most  people would think that this is about the Docker Py library which is for managing your  containers through Python code but that's not the case in this talk.  Now, so why did I decide to talk about this and how did we came up with this solution  for containerizing Jenkins?  Now when I first came to Red Hat I was given the task of replicating my Jenkins job, our  testing framework on my local environment and for doing so I was given a six-page document  that it actually took me approximately two weeks to complete and then when I realized  how big of a fool's errand it was to try and do so plus I also realized that most of the  people on my team was having issues when trying to replicate their environments as well but  at the same time those environments that were successfully working at some point with the  constant evolution of our testing framework, those setups were easily lost.  So there had to be a way to keep up with this and have it centralized and make sure that  no one had to waste two weeks of their time to set it up.  Now at the time we were not using virtual environments therefore you had to install all the requirements  from the testing framework on your local machine and that could potentially cause some dependency  issues on the libraries that you already had installed on your machine.  Now also like the biggest benefit of this was to be able to put a Docker image on an internal  registry that you can run on your organization so that whenever a newcomer started working  just by pulling the image would be able to start debugging straight away.  Now how many of you have been using PyTest on Jenkins already?  Okay, that's a good number.  So there's a couple of things that you need to take into consideration at the time of  trying to replicate this Jenkins environment or this job run.  So first thing, if the job that you're trying to replicate is PyTest you should look at  the PyTest options on your job, on the configuration of your job and here you can have all sorts  of things and these parameters can be used by Jenkins in many other places during the  job execution.  In the case that I will show you in this small sample scenario, we only have a parameter  that will be used on one of the build steps that I will show you further on.  Now you might also have distributed tests and you could also have parameters for determining  to which hosts you want or on which hosts you want the job to run.  So that's one thing to take into consideration.  And if the job itself has any Git integration and if you're actually this is the most common  practice so you probably do.  So it's important to check that all those repositories are going to be used for our local  environment.  So this is really important and we will actually link all those repositories to volumes on our  containers.  Now finally I would look at the build steps on the configuration of the Jenkins job and  it will depend on the complexity of your testing framework but you might have in this case  I just have one build step where it's actually a dynamic bar script that takes one of the  parameters that I defined before for choosing which tests to run.  And you might have additional build steps that could, for example, in our case, generate  configuration files that are used by custom plugins and you need to make sure that all  those steps are covered before you decide to run PyTest on your container.  Now with PyTest virtually any Python module can be registered as a plugin.  So how PyTest does the plugin discovery, how it checks all the plugins that are on your  testing framework, it first starts by loading all the built-in plugins that were installed  with PyTest.  Secondly it looks for, it actually loads all the plugins that are registered through setup  tools entry point.  I will show you a small sample on the next slide where I am actually passing one plugin  that is outside our testing framework through the setup tools.  Third thing that it does is pre-scans the command line for the minus p name option and  it loads the specified plugin before doing the actual command line parsing of PyTest.  In this case you can also choose to deactivate already registered plugins by prefixing the  minus p option.  Actually I didn't expect this to show like that but it's minus p, no semicolon and the  name of the plugin.  So that's how you avoid loading plugins that perhaps are causing issues on your testing  framework.  Next thing that PyTest loads, it looks for all the comftest.py files as inferred by the  command line invocation and if you don't have any test paths specified, it uses the current  directory as a path.  Now you need to note that PyTest does not find comftest.py files in deeper nested subdirectories  at tool start-up.  So it is usually a good idea to keep your comftest.py files in the top level of your  root project.  Now next it does also load recursively all the plugins specified by the PyTest underscore  plugins variable that you can specify inside your comftest.  So that is actually the last place where PyTest looks for custom plugins.  So now if you want to make your plugin externally available, you may define what is so-called  entry point for your distribution so that PyTest finds your plugin module.  Entry points are a feature that is provided by setup tools and PyTest looks specifically  for the PyTest 11, I cannot see now here, it looks for this entry point and it actually  looks for this entry point and it loads all the plugins that you have defined under this  entry point.  So now when I first started to try to containerize this environment, I started by creating a base  image with Fedora and a couple of instructions that I already knew that had to be there.  For example, one of the first things that I had to do was at the time of linking my local  repositories to volumes inside my container, I realized that I had to include all these  repositories into my Python path.  So I managed to do that by including the M instruction on the Docker file for creating  the image.  And later on, I started trying to install all the requirements that were defined on our  testing framework requirements.txt file inside the root directory of the testing framework.  And it was then when I realized that some of the requirements that I was trying to install  with PIP had header files from RPMs that were missing or were not included on any place on  my testing framework.  Because all these dependencies were either already included in the operating system that  was being used for running these tests or being provisioned by Foreman or other services  that were invisible to my testing framework.  Also how I managed to see which plugins were loaded by PyTest at every point while I was  trying to containerize these was by using the trace config option from PyTest which  actually shows you all the plugins that are being recognized by PyTest at the time.  Now as you can see there, I have this run statement that I included on my Docker file.  Now it is recommended that you minimize the number of run commands because in that way  you, it makes for less overhead at the time of actually running your container.  So now when you are leaking a volume on your container to local repo directory on your  machine, if you have the SE Linux policy enforced, then the container won't be able to write  to this directory.  And in the case that it might be possible that your tests are generating log files that  are saved on that same repository.  And if you have SE Linux enforced, the container will fail with permission denied or an IBC  message in the host syslog.  So to avoid that, you need to change the context of the directory, in this case the repository  directory to that tag in particular, sv sandbox file t.  Now this has become much more easier since Docker 1.7 because as you can see on the second  piece of code here, I am preparing the set at the end.  And since Docker 1.7, this statement automatically changes the context of the directory that  you are trying to link to your container.  Therefore, the container will be able to write to that repository.  Now further on, at the very end of my Docker file, I define these two instructions, the  entry point and the cmd.  Now the entry point, it virtually makes your container executable.  How so?  Because when you run your container image, the first command that the container will  run will be the entry point.  And the cmd, it's any additional options that you would like to run with your entry point.  But why is it separated like this?  It's because the cmd is overwriteable.  You can overwrite that by adding options to your run statement for when you want to run  your container.  And therefore, it makes it more flexible.  Now at the time of debugging your PyTests, you might want to, I'm not sure, I use IPDB,  for example, but unless you use PDB or PyTests on PDB, you won't be able to debug because  PyTests is capturing the standard output.  So you need to run PyTests with the minus S option so that you are able to use any other  debuggers apart from PDB or PyTests PDB.  So I went through that real fast.  Now I will show you a little bit of a live demo and a small scenario that I made with  Jenkins.  You can find the testing framework that we are going to be using on this live demo on  this repository.  And the second repository contains the Docker files and the Docker compose file that I am  using for instantiating these Jenkins environment.  So I'm not sure if I'm able to see here.  So this is Jenkins instance that I have running on a container, of course, in my local machine.  And this job in particular is a parameterized job that takes only one parameter and based  on the input that we put there, it will run one test or the other as we saw on the bash  script before.  So I will run a bad test and a good test.  One should fail, the other one shouldn't.  Most likely both will fail.  So we got a failing test here.  It's success.  Let's say I want to debug this failing test that is running on my Jenkins instance.  Now I already created a container image that will replicate my Jenkins instance and a container  that I can use for debugging my Python locally.  Now I have some scripts here.  So if I run this script for running this bad test, I can show you what does this file contain.  It's basically a run instruction for Docker to run with the FC24 container that I created  with this Docker file.  And basically I am telling straight away to PyTest which test to run.  So now in theory, if I...  This is the testing framework.  So in theory, if I edit my bad test to include this breakpoint, then I should be able now  to run my container and I should be able to hit that breakpoint.  And yeah, there you see.  So you can...  Why don't you run on one?  Because that actually doesn't...  It actually hits the breakpoint when an exception is raised.  So it actually stops the execution whenever a trace bug is raised.  Now you might have no failing tests that you will still want to debug.  So...  Okay.  So let me show you...  I got another script here as well.  And let me show you...  I have defined on my Conf test, I have defined a small plugin, a small custom plugin that  is in the form of a mark.  And I am using that on my good file.  No, sorry.  So basically this mark is for including an option name on the PyTest run that will look  for any tests that are tagged with whatever I introduce on my PyTest run.  So if I go here...  Now...  Yeah.  So if I run the good file now, this file in fact has...  As you can see, I am passing the minus E option and the good tab that was defined on my marker  for that test in particular.  So if I try to run this container and I change the...  I change the label to whatever, it will actually skip the test because it's not the label for  good that we had defined on that test.  Now additionally, we can see with the trace config, this is also just running PyTest with  the trace config.  On the trace config you will see all the planes that have been registered by PyTest as well  as the comftests that have been loaded by...  So if I do this, you can see...  In this case, it's showing me that it has loaded the comftest from the root directory  which is on the container itself because I am routing my volumes...  I am linking my volumes to the main root folder on my container.  So that's pretty much it.  Now...  Yeah.  So, yeah, that's pretty much it.  Now if you have any questions, I'm afraid we don't have any more time.  Do we?  Okay.  Yeah, sure.  I'm sorry, man.  For this container in particular?  Yeah.  So this is the one for the Jenkins instance that it's actually picking up the base image  from Docker Hub, this Jenkins latest.  And I include some other instructions for what I needed at the time.  Then you get the Jenkins slave Docker file which is just the Java container, base container.  And there I am downloading the Java agent from Jenkins for executing the file then.  So you can find this on GitHub.  There is...  You can find the Docker files there and...  I should be...  The Docker files where you had entry points and stuff, where are they?  Ah, well, for this talk in particular, I didn't use any entry points or CMD, but it's  like that.  It's just like you can only use CMD once and the entry point should be your default command  to be executed with the Docker.  Hello.  Thank you for the presentation.  It was a very nice setup.  Sorry, I cannot hear you.  Thank you for the presentation.  That was a nice setup.  I have two questions.  First of all, this looks like one of the benefits of this might be able to run multiple tests  from Jenkins at the same time.  Is this something that you've tried and it works properly with the setup?  Yeah, you can run any test that you have on your testing framework if you already have  it set up like that on your Jenkins instance.  So this Jenkins instance was just queuing them because that's how Jenkins was set up?  This Jenkins instance was just to show the two different places where the Jenkins job  that is running the PyTest and my local PyTest run that will be able to debug the test.  Okay, cool.  My next question, can you tell me what that shell you had, it showed whether the git was  dirty or clean?  It looked very nice.  I liked it.  This is all my set SH with some tweaks.  Okay, thank you.  Hello.  Thank you for the talk.  It's very important.  My question is when it comes to deploying your application, I can understand the benefits  of using Docker.  There are a lot of advantages.  Like encapsulating all of the businesses inside your containers so you can ship it as  is or make your application cloud native ready, something like that.  But talking about testing only, what is the real advantage of using the containerized  solution with respect to other available solutions that in my opinion might be more  lightweight solutions like virtual environment?  So have you considered this other solution?  What is your opinion?  As I said before, given the complexity of our testing framework, this was the best solution  at the time.  And as I said also, I don't want to preach or sweet talk you into using Docker or containers  for replicating your Jenkins instance.  But this is actually saving us a lot of time for newcomers and also it's maintaining some  sort of uniformity on our testing framework.  Do we have any more questions?  No.  Thank you very much Gonzalo.  Thank you Gonzalo once more.
Gonzalo_Rafuls_-_Dockerized_pytests_iEKsAxUPDYY_0_275.png Gonzalo_Rafuls_-_Dockerized_pytests_iEKsAxUPDYY_276_14879.png Gonzalo_Rafuls_-_Dockerized_pytests_iEKsAxUPDYY_14880_27635.png Gonzalo_Rafuls_-_Dockerized_pytests_iEKsAxUPDYY_27636_29135.png Gonzalo_Rafuls_-_Dockerized_pytests_iEKsAxUPDYY_29136_29339.png Gonzalo_Rafuls_-_Dockerized_pytests_iEKsAxUPDYY_29340_31451.png Gonzalo_Rafuls_-_Dockerized_pytests_iEKsAxUPDYY_31452_31835.png Gonzalo_Rafuls_-_Dockerized_pytests_iEKsAxUPDYY_31836_32255.png Gonzalo_Rafuls_-_Dockerized_pytests_iEKsAxUPDYY_32256_33983.png Gonzalo_Rafuls_-_Dockerized_pytests_iEKsAxUPDYY_33984_34235.png Gonzalo_Rafuls_-_Dockerized_pytests_iEKsAxUPDYY_34236_34487.png Gonzalo_Rafuls_-_Dockerized_pytests_iEKsAxUPDYY_34488_34559.png Gonzalo_Rafuls_-_Dockerized_pytests_iEKsAxUPDYY_34560_35075.png Gonzalo_Rafuls_-_Dockerized_pytests_iEKsAxUPDYY_35076_35555.png Gonzalo_Rafuls_-_Dockerized_pytests_iEKsAxUPDYY_35556_35627.png Gonzalo_Rafuls_-_Dockerized_pytests_iEKsAxUPDYY_35628_36047.png Gonzalo_Rafuls_-_Dockerized_pytests_iEKsAxUPDYY_36048_36095.png Gonzalo_Rafuls_-_Dockerized_pytests_iEKsAxUPDYY_36096_36383.png Gonzalo_Rafuls_-_Dockerized_pytests_iEKsAxUPDYY_36384_38315.png Gonzalo_Rafuls_-_Dockerized_pytests_iEKsAxUPDYY_38316_38783.png Gonzalo_Rafuls_-_Dockerized_pytests_iEKsAxUPDYY_38784_39659.png Gonzalo_Rafuls_-_Dockerized_pytests_iEKsAxUPDYY_39660_39719.png Gonzalo_Rafuls_-_Dockerized_pytests_iEKsAxUPDYY_39720_39875.png Gonzalo_Rafuls_-_Dockerized_pytests_iEKsAxUPDYY_39876_39923.png Gonzalo_Rafuls_-_Dockerized_pytests_iEKsAxUPDYY_39924_41351.png Gonzalo_Rafuls_-_Dockerized_pytests_iEKsAxUPDYY_41352_42071.png Gonzalo_Rafuls_-_Dockerized_pytests_iEKsAxUPDYY_42072_42119.png Gonzalo_Rafuls_-_Dockerized_pytests_iEKsAxUPDYY_42120_42179.png Gonzalo_Rafuls_-_Dockerized_pytests_iEKsAxUPDYY_42180_42323.png Gonzalo_Rafuls_-_Dockerized_pytests_iEKsAxUPDYY_42324_42399.png
Thank you all guys for coming. I wasn't expecting this many people, but that's a good sign, I guess. First let me start a little bit by introducing myself. My name is Gonzalo Raffuls and I do quality engineering for Red Hat in Czech Republic for its virtualization solution which is over on the AppStream. So during this talk I will try to go fast and make it light, it's been a long week. To begin with I'm going to be talking a little bit about, I'm going to actually show you a simple test scenario for running PyTest on Jenkins and we're going to have a little bit of a high overview on PyTest custom plugins and how PyTest loads custom plugins. Then we will look a little bit at the challenges faced at the time of trying to replicate this Jenkins job on your local environment with Docker. We will look also at Docker as a solution for doing this and then if we have enough time I'll make a small live demo if it doesn't fail miserably. Now I'm not going to be talking, I'm not going to be, I don't want to intend to sweet talk you and to start using containers, you should be aware of the security implications that comes with containerizing your applications. We're not going to talk about containerizing Jenkins, that's already been done. It's ideal if you already have some high overview on Jenkins, PyTest and Docker and also most people would think that this is about the Docker Py library which is for managing your containers through Python code but that's not the case in this talk. Now, so why did I decide to talk about this and how did we came up with this solution for containerizing Jenkins? Now when I first came to Red Hat I was given the task of replicating my Jenkins job, our testing framework on my local environment and for doing so I was given a six-page document that it actually took me approximately two weeks to complete and then when I realized how big of a fool's errand it was to try and do so plus I also realized that most of the people on my team was having issues when trying to replicate their environments as well but at the same time those environments that were successfully working at some point with the constant evolution of our testing framework, those setups were easily lost. So there had to be a way to keep up with this and have it centralized and make sure that no one had to waste two weeks of their time to set it up. Now at the time we were not using virtual environments therefore you had to install all the requirements from the testing framework on your local machine and that could potentially cause some dependency issues on the libraries that you already had installed on your machine. Now also like the biggest benefit of this was to be able to put a Docker image on an internal registry that you can run on your organization so that whenever a newcomer started working just by pulling the image would be able to start debugging straight away. Now how many of you have been using PyTest on Jenkins already? Okay, that's a good number. So there's a couple of things that you need to take into consideration at the time of trying to replicate this Jenkins environment or this job run. So first thing, if the job that you're trying to replicate is PyTest you should look at the PyTest options on your job, on the configuration of your job and here you can have all sorts of things and these parameters can be used by Jenkins in many other places during the job execution. In the case that I will show you in this small sample scenario, we only have a parameter that will be used on one of the build steps that I will show you further on. Now you might also have distributed tests and you could also have parameters for determining to which hosts you want or on which hosts you want the job to run. So that's one thing to take into consideration. And if the job itself has any Git integration and if you're actually this is the most common practice so you probably do. So it's important to check that all those repositories are going to be used for our local environment. So this is really important and we will actually link all those repositories to volumes on our containers. Now finally I would look at the build steps on the configuration of the Jenkins job and it will depend on the complexity of your testing framework but you might have in this case I just have one build step where it's actually a dynamic bar script that takes one of the parameters that I defined before for choosing which tests to run. And you might have additional build steps that could, for example, in our case, generate configuration files that are used by custom plugins and you need to make sure that all those steps are covered before you decide to run PyTest on your container. Now with PyTest virtually any Python module can be registered as a plugin. So how PyTest does the plugin discovery, how it checks all the plugins that are on your testing framework, it first starts by loading all the built-in plugins that were installed with PyTest. Secondly it looks for, it actually loads all the plugins that are registered through setup tools entry point. I will show you a small sample on the next slide where I am actually passing one plugin that is outside our testing framework through the setup tools. Third thing that it does is pre-scans the command line for the minus p name option and it loads the specified plugin before doing the actual command line parsing of PyTest. In this case you can also choose to deactivate already registered plugins by prefixing the minus p option. Actually I didn't expect this to show like that but it's minus p, no semicolon and the name of the plugin. So that's how you avoid loading plugins that perhaps are causing issues on your testing framework. Next thing that PyTest loads, it looks for all the comftest.py files as inferred by the command line invocation and if you don't have any test paths specified, it uses the current directory as a path. Now you need to note that PyTest does not find comftest.py files in deeper nested subdirectories at tool start-up. So it is usually a good idea to keep your comftest.py files in the top level of your root project. Now next it does also load recursively all the plugins specified by the PyTest underscore plugins variable that you can specify inside your comftest. So that is actually the last place where PyTest looks for custom plugins. So now if you want to make your plugin externally available, you may define what is so-called entry point for your distribution so that PyTest finds your plugin module. Entry points are a feature that is provided by setup tools and PyTest looks specifically for the PyTest 11, I cannot see now here, it looks for this entry point and it actually looks for this entry point and it loads all the plugins that you have defined under this entry point. So now when I first started to try to containerize this environment, I started by creating a base image with Fedora and a couple of instructions that I already knew that had to be there. For example, one of the first things that I had to do was at the time of linking my local repositories to volumes inside my container, I realized that I had to include all these repositories into my Python path. So I managed to do that by including the M instruction on the Docker file for creating the image. And later on, I started trying to install all the requirements that were defined on our testing framework requirements.txt file inside the root directory of the testing framework. And it was then when I realized that some of the requirements that I was trying to install with PIP had header files from RPMs that were missing or were not included on any place on my testing framework. Because all these dependencies were either already included in the operating system that was being used for running these tests or being provisioned by Foreman or other services that were invisible to my testing framework. Also how I managed to see which plugins were loaded by PyTest at every point while I was trying to containerize these was by using the trace config option from PyTest which actually shows you all the plugins that are being recognized by PyTest at the time. Now as you can see there, I have this run statement that I included on my Docker file. Now it is recommended that you minimize the number of run commands because in that way you, it makes for less overhead at the time of actually running your container. So now when you are leaking a volume on your container to local repo directory on your machine, if you have the SE Linux policy enforced, then the container won't be able to write to this directory. And in the case that it might be possible that your tests are generating log files that are saved on that same repository. And if you have SE Linux enforced, the container will fail with permission denied or an IBC message in the host syslog. So to avoid that, you need to change the context of the directory, in this case the repository directory to that tag in particular, sv sandbox file t. Now this has become much more easier since Docker 1.7 because as you can see on the second piece of code here, I am preparing the set at the end. And since Docker 1.7, this statement automatically changes the context of the directory that you are trying to link to your container. Therefore, the container will be able to write to that repository. Now further on, at the very end of my Docker file, I define these two instructions, the entry point and the cmd. Now the entry point, it virtually makes your container executable. How so? Because when you run your container image, the first command that the container will run will be the entry point. And the cmd, it's any additional options that you would like to run with your entry point. But why is it separated like this? It's because the cmd is overwriteable. You can overwrite that by adding options to your run statement for when you want to run your container. And therefore, it makes it more flexible. Now at the time of debugging your PyTests, you might want to, I'm not sure, I use IPDB, for example, but unless you use PDB or PyTests on PDB, you won't be able to debug because PyTests is capturing the standard output. So you need to run PyTests with the minus S option so that you are able to use any other debuggers apart from PDB or PyTests PDB. So I went through that real fast. Now I will show you a little bit of a live demo and a small scenario that I made with Jenkins. You can find the testing framework that we are going to be using on this live demo on this repository. And the second repository contains the Docker files and the Docker compose file that I am using for instantiating these Jenkins environment. So I'm not sure if I'm able to see here. So this is Jenkins instance that I have running on a container, of course, in my local machine. And this job in particular is a parameterized job that takes only one parameter and based on the input that we put there, it will run one test or the other as we saw on the bash script before. So I will run a bad test and a good test. One should fail, the other one shouldn't. Most likely both will fail. So we got a failing test here. It's success. Let's say I want to debug this failing test that is running on my Jenkins instance. Now I already created a container image that will replicate my Jenkins instance and a container that I can use for debugging my Python locally. Now I have some scripts here. So if I run this script for running this bad test, I can show you what does this file contain. It's basically a run instruction for Docker to run with the FC24 container that I created with this Docker file. And basically I am telling straight away to PyTest which test to run. So now in theory, if I... This is the testing framework. So in theory, if I edit my bad test to include this breakpoint, then I should be able now to run my container and I should be able to hit that breakpoint. And yeah, there you see. So you can... Why don't you run on one? Because that actually doesn't... It actually hits the breakpoint when an exception is raised. So it actually stops the execution whenever a trace bug is raised. Now you might have no failing tests that you will still want to debug. So... Okay. So let me show you... I got another script here as well. And let me show you... I have defined on my Conf test, I have defined a small plugin, a small custom plugin that is in the form of a mark. And I am using that on my good file. No, sorry. So basically this mark is for including an option name on the PyTest run that will look for any tests that are tagged with whatever I introduce on my PyTest run. So if I go here... Now... Yeah. So if I run the good file now, this file in fact has... As you can see, I am passing the minus E option and the good tab that was defined on my marker for that test in particular. So if I try to run this container and I change the... I change the label to whatever, it will actually skip the test because it's not the label for good that we had defined on that test. Now additionally, we can see with the trace config, this is also just running PyTest with the trace config. On the trace config you will see all the planes that have been registered by PyTest as well as the comftests that have been loaded by... So if I do this, you can see... In this case, it's showing me that it has loaded the comftest from the root directory which is on the container itself because I am routing my volumes... I am linking my volumes to the main root folder on my container. So that's pretty much it. Now... Yeah. So, yeah, that's pretty much it. Now if you have any questions, I'm afraid we don't have any more time. Do we? Okay. Yeah, sure. I'm sorry, man. For this container in particular? Yeah. So this is the one for the Jenkins instance that it's actually picking up the base image from Docker Hub, this Jenkins latest. And I include some other instructions for what I needed at the time. Then you get the Jenkins slave Docker file which is just the Java container, base container. And there I am downloading the Java agent from Jenkins for executing the file then. So you can find this on GitHub. There is... You can find the Docker files there and... I should be... The Docker files where you had entry points and stuff, where are they? Ah, well, for this talk in particular, I didn't use any entry points or CMD, but it's like that. It's just like you can only use CMD once and the entry point should be your default command to be executed with the Docker. Hello. Thank you for the presentation. It was a very nice setup. Sorry, I cannot hear you. Thank you for the presentation. That was a nice setup. I have two questions. First of all, this looks like one of the benefits of this might be able to run multiple tests from Jenkins at the same time. Is this something that you've tried and it works properly with the setup? Yeah, you can run any test that you have on your testing framework if you already have it set up like that on your Jenkins instance. So this Jenkins instance was just queuing them because that's how Jenkins was set up? This Jenkins instance was just to show the two different places where the Jenkins job that is running the PyTest and my local PyTest run that will be able to debug the test. Okay, cool. My next question, can you tell me what that shell you had, it showed whether the git was dirty or clean? It looked very nice. I liked it. This is all my set SH with some tweaks. Okay, thank you. Hello. Thank you for the talk. It's very important. My question is when it comes to deploying your application, I can understand the benefits of using Docker. There are a lot of advantages. Like encapsulating all of the businesses inside your containers so you can ship it as is or make your application cloud native ready, something like that. But talking about testing only, what is the real advantage of using the containerized solution with respect to other available solutions that in my opinion might be more lightweight solutions like virtual environment? So have you considered this other solution? What is your opinion? As I said before, given the complexity of our testing framework, this was the best solution at the time. And as I said also, I don't want to preach or sweet talk you into using Docker or containers for replicating your Jenkins instance. But this is actually saving us a lot of time for newcomers and also it's maintaining some sort of uniformity on our testing framework. Do we have any more questions? No. Thank you very much Gonzalo. Thank you Gonzalo once more.
over on the AppStream. So during this talk I will high overview on PyTest custom plugins and how little bit at the challenges faced at the time of having issues when trying to replicate their I first started to try to containerize this with Fedora and a couple of instructions that I flexible. Now at the time of debugging your instantiating these Jenkins environment. So I'm that I can use for debugging my Python locally. this, you can see... In this case, it's showing run multiple tests from Jenkins at the same time.