How openQA works
Kulow, Stephan
openQA has become core part of both openSUSE and SUSE development (and beyond), but there are many myths what it actually does. This talk targets interested developers who used openQA before, but used the test API as blackbox so far.
 Hi. This, my talk is about OpenQA and how it works in the background. My plan was to have this in the line of OpenQA talks that piled up from beginners to experts. I somehow ended up being the first talk at all in the conference. So let's see how this works out. About me, I'm Stefan Kullo. I worked for SUSE for a long time. I was the release manager of Open SUSE until the release of Leap42.1. And currently I'm acting as OpenQA architect and test automation champion or however you want to name it. In my private life, I have two kids that I'm trying to teach free software. No, I don't. This talk will be about things you don't yet, you might not yet know about OpenQA, the architecture on how the whole system works internally and implementation details that would, if you know them, make you a better test driver in OpenQA. So you see, this is already some, I require some knowledge about OpenQA. So I will not have a beginner's guide on how to write tests. And I will not, yeah, you can still stay here if you don't know shit about OpenQA. I tried to have some introduction, but I might lose you quickly. So let's see. So OpenQA started in 2010 by Bernhard Wiedemann, our camera kind, to create videos from Open SUSE installations. And the name, EZO2Video, still sticks from that time. End of 2011 was the first release of OpenQA. You can still find it in the archives of news OpenSuser.org. And, but back then it was still mainly a tool for Bernhard and some selected others to create better bug reports. And this is how it looked like. So it was, if you know OpenQA now, it is not so much different. And actually some of the features that are on the left are we are currently re-implementing to track closed bug reports. So this is how it currently looks like. So it's still about GNOME and KDE tests, if you see, if you compare. It's very hard to read, but trust me, it's about desktop tests. Oh, good. The architecture is nowadays mainly about one central web interface, which contains a classic web application that has HTTP and a REST interface for an API. And all users use this as an entrance point to the whole system. There's some scheduler. There's some scheduler that keeps the jobs rotated between the workers and there's a database. And all these three components communicate through debas within one VM that I labeled here, WebUI. And then you have what I tried to express on the right, several workers that connect to this service, to this service, and take jobs and run the OpenQA tests. One of these workers I zoomed in at the bottom. You have several instances of a process called either-to-video running. This one splits into three in between. We have one test runner that actually runs the Perl tests. You have one commands server that takes commands from the outside, including the system under test. And the WebUI, in case you're using the interactive mode of the WebUI. And then there is the actual backend, which can fork several other processes, including QAMU or Lipvert commands. Or in case of S390 support an X3270 term or IPMI commands. So the backends are actually the real... I really should slow down. I was worried that I would have too many slides, but the backends are the real troublemakers. They share some common properties. One of them is that they can talk VNC. One of them is that they are able to power management the system under test. In QAMU this is really easy because you can just start and stop QAMU. But if you're talking about an S390 guest or an IPMI controlled hardware metal, this is more complicated. And the third one is in our case that we want to have access to the serial console to be able to have better console tests. The main backend still use is QAMU using KVM. And this QAMU one can emulate a lot of hardware setups. We have, for example, we have one case where we emulate in specific Dell laptop. We can have UEFI BIOS, we have legacy BIOS, we have several IOPAS systems. We have IDE SCSI, we have NVMe lately. We can emulate a RAID system, software RAID, we can have multi-path and so on. We can all emulate this just with QAMU. Now let me continue talking about backends for a while. But we also have other backends that, for example, connect to an lib-vert domain or host through SSH. So it can be actually everything. We have Hyper-V support through a VNC hop implemented. We have some S390 already mentioned. And these are all invisible to the test driver because the tests do the very same thing. Unless you're very specific, for example, reboot, but all the other stuff is done through a very generic API. And one core component is the needle matching that I would like to dig into further. A needle is a region of interest in a reference screen. Most of you that are right in the stock will know that already. But to give you an example, this is actually the pop-up that happens when you press F6, I think, on an OpenZoozer installation. It gives you a way to edit the HTTP installation URL. But this is actually only a zoom in. And the web UI will actually gray out everything else. That is not of interest. So let's give you a more detailed example. This, for example, is an quite old KDE screenshot with an X term started. So this is, we have the test driver wanted to verify that the X term actually started. So we had a reference screen. And this reference screen contains a region of interest. And that is this region that is not grayed out here. This is the part that identifies this screen as the X term has finished starting. The rest is not of interest. So this is the needle. This is what we call the needle. So if I go now on another screen where you don't have KDE, but still there's an X term started, we are asking OpenQA to assert that there's an X term started by means of having a third screen X term. And OpenQA would look at this screen and the first thing it does is it grays out the screen to 16 shades of gray and blurs the whole thing. I'm not sure how visible the difference is from behind, but this is the clear picture and this is the blurred picture. I guess from behind it's not obvious, but it's... So basically OpenQA looks at the screen as a very old snake. It's not... Some people think that OpenQA compares pixels by pixels, but it's not even close to that. The next thing it will do is look at this great, great out blurred screen to identify the most likely spot that the needle is. So this is some brain dump of OpenQA, let's say. The wider the pixel is, the more likely it contains the region of interest. What is invisible to the human eye is that it's actually exactly there where we headed the highest value. And what we are doing with this information is we're going through the likely spots and evaluating the actual pixel diff of this 16 shades of gray. And the number of differences we see in this area is counted and weighted. So we are not taking every error as an error, but we look on how big the area was and how many differences we saw. So the larger the area, the more forgiving OpenQA is to differences. And then we have some, let's say, magic formula that decides how many percentages this correlates to. This formula has grown over the time because we changed it, we changed the algorithm after we had tons of needles. So we had to adopt the formula to look like the old system, but it was still, give out similar match numbers. But in this very X term example, so you have Bernhardt on the screen, always, actually. On the left you have the KDE screenshot, on the right you have the GNOME screenshot, only blurred, grayscaled, the region of interest. And the brain dump of OpenQA says it's completely black. So the difference is exactly zero at every pixel, so it's 100% match. Any questions so far? Good. So what I think that is inherited from here is that big changes are obviously problematic, but what is not so obvious is that small changes to layouts can create quite some noise in the pixel diffs. The one prominent example is if we change the fonts or the font weighting or the font spacing or you name it, the difference to the user is minimal. It is just the L written differently or the S moves closer to the T or it's very tiny changes most often, but many pixels change because the follow up letters move to, if the S moves closer to the T. So this is in 12.3 screenshot. And if I flip the fourth and back, you will ignore the left side, but only concentrate on the license part. You might spot the difference if I do this, but if I asked you which is which, it's already getting complicated. But to OpenQA, does it difference? Here we have the same exercise again on the left side before, right side after. And now I was not really sure how it is visible, but the pixel difference at the bottom contains quite some noise. Is this visible? I see nodding. So this is a 34% match. This is how it looks like on OpenQA internally. Okay. The S-Hertz green is for gruy testing or for installation tests, the most common call actually, but it is important to understand that tests, the test code is blocking code. So in your test, you do not care what happens to the system under test. You're only writing the S-Hertz or the commands. And in between the system under test and QAMU and the backend, all of them continue running, but the tests actually block. So you do an S-Hertz screen on X-Term. This might take a while depending on what happened else or you do a S-Hertz screen system booted and then you reboot and then a S-Hertz screen waiting for the booted desktop. And this can take minutes, but in between this call will just block. So after you have the assertion that the X-Term is there, you type some string. You don't care how long this takes and what it does. And in the end, you do a S-Hertz screen on Hello-Type, which means once the string in the font is rendered in the font you're expecting, it will actually pass. All these three commands are blocking calls. So what happens in the background, if I jump back to the architecture that I should have copied, easy to video, we'll get the command from the test runner that there's supposed to be an assert screen. But easy to video will pass some commands to the backend to prepare the reference screen on the needles matching to it and then will continuously polling the backend if it found something in the current screen shot. And only when the backend says there's a timeout or there is a match found, it will pass back to the test runner the result. And in between, it will await commands from that what you see as commands there in case the web UI decided to inject new needles or the testers had stopped on the web UI and all this happens in the easy to video program. This is what I named here as an event loop. And the actual grabbing the screen from VNC, the actual check the needles database, check the needle comparison that we just saw, all this happens in the backend process. And the backend process is also the one that has to take care that we grab a video, a screenshot regularly from the VNC and pipe it into the video so that we are later on can have a regularly continuous video so that every frame also is a repeated number of time. So I want actually the last thing I have is our consoles. Cables are a very important concept to support other platforms. When we had only QEMU, it was QEMU like simulations. It was really easy to just have send key alt F2 to switch to another console or open an X term or so we had all these send keys that are very Linux specific or QEMU specific even. But now we have consoles that you can drive through the test API. So you do select console, user console and depending on the platform you are on, you will either switch to alt F2, Linux console or to some X term that has some SSH connection open to the system under test or the other console would be root console or we have lock console or we have tons of consoles. And the QEMU consoles that we have are all so called TTY consoles. They are bound to one specific Linux terminal. But they are only second most used. The one that is most prominent in QEMU test is called SUT and this is the one that actually talks VNC to QEMU. So this is the one that does the assert screen. This is the one that will type string into the currently selected TTY console and so on. And that's what brings us to the last point on the slide that these consoles can be stacked. So we have a TTY console that is actually connected to the SUT console because if you do type string it will type into the TTY console but actually use the SUT console to do this over VNC. And there are other examples where we have this X3 270 terminal that is running in some local X VNC. There's also a console and so on. Really fast shit. The example I just mentioned is where we have X terms running locally on the worker as part of the backend process that will actually drive commands on another machine that you can't see because it's somewhere remote and we don't have a video of it. But a local X VNC makes it possible that we can actually talk VNC to that console as well. So we can see what is typed locally in this terminal and then have a video of it and we can assert screen but when we do assert screens on this local X term running the commands on a remote machine it's more complicated. That's why I put it at the end. But the key thing is that the tests that you write are very, very similar to what you had for QEMO. So when we introduced this IPMI backend to test real hardware we had to change one test that was actually booting the system and after that we could use the same installation tests that we had before that were asset screening, send type and so on. And when we introduced or when we worked on S390 support it was a bit more complicated especially as we had to introduce the console handling at all. But in the end most of the tests are shared between all platforms. So this is some benefit that we have now. We can test let's say almost every hardware with the same set of tests as long as we have SSH connection to it and we can power cycle it. This is the VNC part that I mentioned initially for having a common factor between the back ends is lost then but we will use some local X term to have the screen but for we can boot the machines remotely, we can install them remotely, we can test things on them remotely. This is all things we can drive so we don't need a lot from the hardware to be tested as long as we don't need to see that it's video. But for servers this is the perfect match and this brings even more test capabilities to the system. Good. This leaves us five minutes for answers. Any questions for that? I think someone has to press here. Test, test. I have a question regarding comparing needles. You described the algorithm and my question is probably this task comparing two images is solved by many different ways. This algorithm is something which is developed inside the open QA or it's some reuse of some existing algorithm. And in both cases if you answer yes or no why you select this way of doing comparison? I must admit I did not really understand the question partly because of the audio and partly because of your pronunciation. The question is the algorithm which you are using to comparing needles. This algorithm is something generic used outside open QA2 or it's something developed inside open QA for solving this problem? The algorithm that we use to have to do this, this is basically the backbone of the whole needle matching. This is based on open CVs template matching. If you Google template matching you will find other examples of people using this. For example automated autonomous test cars driving. They also do something very similar to detect traffic signs. When you drive they have all these cameras in your cars and they will detect traffic signs just as we are detecting needles. A bit more complicated because they also have to adopt to having different viewpoints. So if you are coming from this side of the traffic sign you still have to detect it then if you come from this side. But it's basically the same algorithms there. Thank you. Hi. I must admit that I don't have very deep knowledge with open QA but wouldn't it be possible to use something like OCR to detect fonts to circumvent the pixel matching? Do you know what I mean? The fact is we did some experiments with it. We started with that and we continued with that. But the fact is that OCR is just as unreliable as this thing is. If you have to train the OCR engine for every font change you have the same problem there than we have right now. The advantage of OCR would be that for example during installation you have tons of text scrolling through and there's no really good way to detect an error for example. So a warning that you want to assert that there's no warning scrolling through. And if you had OCR you could actually make this happen. But the problem is as long as OCR is so unreliable as it is right now or most likely ever will be if you ask me. The chances that you will miss these warnings is just as slim as currently. So that's why OCR sounds reasonable but it's not so well suited in practice. For console tests we don't do it at all and for GUI testing we don't care about the text so much. So that OCR is a lot of investment and there is support for it. But whenever someone finishes the support or bug fixes it to work again, he soon realizes or she soon realizes it's not really worth it and then it puts it again in the code base. So we use Tesseract for that which is already quite good OCR engine and I use it for private projects where you can train on the font used. For example in my case in a dictionary but for having text in a random screenshot it's just not suited. Thanks. Okay. There are no more questions. I'm done after exactly half an hour. Thank you for your patience.
 Hi.  This, my talk is about OpenQA and how it works in the background.  My plan was to have this in the line of OpenQA talks that piled up from beginners to experts.  I somehow ended up being the first talk at all in the conference.  So let's see how this works out.  About me, I'm Stefan Kullo.  I worked for SUSE for a long time.  I was the release manager of Open SUSE until the release of Leap42.1.  And currently I'm acting as OpenQA architect and test automation champion or however you want to name it.  In my private life, I have two kids that I'm trying to teach free software.  No, I don't.  This talk will be about things you don't yet, you might not yet know about OpenQA,  the architecture on how the whole system works internally and implementation details that would,  if you know them, make you a better test driver in OpenQA.  So you see, this is already some, I require some knowledge about OpenQA.  So I will not have a beginner's guide on how to write tests.  And I will not, yeah, you can still stay here if you don't know shit about OpenQA.  I tried to have some introduction, but I might lose you quickly.  So let's see.  So OpenQA started in 2010 by Bernhard Wiedemann, our camera kind,  to create videos from Open SUSE installations.  And the name, EZO2Video, still sticks from that time.  End of 2011 was the first release of OpenQA.  You can still find it in the archives of news OpenSuser.org.  And, but back then it was still mainly a tool for Bernhard and some selected others to create better bug reports.  And this is how it looked like.  So it was, if you know OpenQA now, it is not so much different.  And actually some of the features that are on the left are we are currently re-implementing to track closed bug reports.  So this is how it currently looks like.  So it's still about GNOME and KDE tests, if you see, if you compare.  It's very hard to read, but trust me, it's about desktop tests.  Oh, good.  The architecture is nowadays mainly about one central web interface,  which contains a classic web application that has HTTP and a REST interface for an API.  And all users use this as an entrance point to the whole system.  There's some scheduler.  There's some scheduler that keeps the jobs rotated between the workers and there's a database.  And all these three components communicate through debas within one VM that I labeled here, WebUI.  And then you have what I tried to express on the right, several workers that connect to this service, to this service,  and take jobs and run the OpenQA tests.  One of these workers I zoomed in at the bottom.  You have several instances of a process called either-to-video running.  This one splits into three in between.  We have one test runner that actually runs the Perl tests.  You have one commands server that takes commands from the outside, including the system under test.  And the WebUI, in case you're using the interactive mode of the WebUI.  And then there is the actual backend, which can fork several other processes, including QAMU or Lipvert commands.  Or in case of S390 support an X3270 term or IPMI commands.  So the backends are actually the real...  I really should slow down. I was worried that I would have too many slides, but the backends are the real troublemakers.  They share some common properties.  One of them is that they can talk VNC.  One of them is that they are able to power management the system under test.  In QAMU this is really easy because you can just start and stop QAMU.  But if you're talking about an S390 guest or an IPMI controlled hardware metal, this is more complicated.  And the third one is in our case that we want to have access to the serial console to be able to have better console tests.  The main backend still use is QAMU using KVM.  And this QAMU one can emulate a lot of hardware setups.  We have, for example, we have one case where we emulate in specific Dell laptop.  We can have UEFI BIOS, we have legacy BIOS, we have several IOPAS systems.  We have IDE SCSI, we have NVMe lately.  We can emulate a RAID system, software RAID, we can have multi-path and so on.  We can all emulate this just with QAMU.  Now let me continue talking about backends for a while.  But we also have other backends that, for example, connect to an lib-vert domain or host through SSH.  So it can be actually everything.  We have Hyper-V support through a VNC hop implemented.  We have some S390 already mentioned.  And these are all invisible to the test driver because the tests do the very same thing.  Unless you're very specific, for example, reboot, but all the other stuff is done through  a very generic API.  And one core component is the needle matching that I would like to dig into further.  A needle is a region of interest in a reference screen.  Most of you that are right in the stock will know that already.  But to give you an example, this is actually the pop-up that happens when you press F6,  I think, on an OpenZoozer installation.  It gives you a way to edit the HTTP installation URL.  But this is actually only a zoom in.  And the web UI will actually gray out everything else.  That is not of interest.  So let's give you a more detailed example.  This, for example, is an quite old KDE screenshot with an X term started.  So this is, we have the test driver wanted to verify that the X term actually started.  So we had a reference screen.  And this reference screen contains a region of interest.  And that is this region that is not grayed out here.  This is the part that identifies this screen as the X term has finished starting.  The rest is not of interest.  So this is the needle.  This is what we call the needle.  So if I go now on another screen where you don't have KDE, but still there's an X term  started, we are asking OpenQA to assert that there's an X term started by means of having  a third screen X term.  And OpenQA would look at this screen and the first thing it does is it grays out the screen  to 16 shades of gray and blurs the whole thing.  I'm not sure how visible the difference is from behind, but this is the clear picture  and this is the blurred picture.  I guess from behind it's not obvious, but it's...  So basically OpenQA looks at the screen as a very old snake.  It's not...  Some people think that OpenQA compares pixels by pixels, but it's not even close to that.  The next thing it will do is look at this great, great out blurred screen to identify  the most likely spot that the needle is.  So this is some brain dump of OpenQA, let's say.  The wider the pixel is, the more likely it contains the region of interest.  What is invisible to the human eye is that it's actually exactly there where we headed  the highest value.  And what we are doing with this information is we're going through the likely spots and  evaluating the actual pixel diff of this 16 shades of gray.  And the number of differences we see in this area is counted and weighted.  So we are not taking every error as an error, but we look on how big the area was and how  many differences we saw.  So the larger the area, the more forgiving OpenQA is to differences.  And then we have some, let's say, magic formula that decides how many percentages this correlates  to.  This formula has grown over the time because we changed it, we changed the algorithm after  we had tons of needles.  So we had to adopt the formula to look like the old system, but it was still, give out  similar match numbers.  But in this very X term example, so you have Bernhardt on the screen, always, actually.  On the left you have the KDE screenshot, on the right you have the GNOME screenshot, only  blurred, grayscaled, the region of interest.  And the brain dump of OpenQA says it's completely black.  So the difference is exactly zero at every pixel, so it's 100% match.  Any questions so far?  Good.  So what I think that is inherited from here is that big changes are obviously problematic,  but what is not so obvious is that small changes to layouts can create quite some noise in  the pixel diffs.  The one prominent example is if we change the fonts or the font weighting or the font  spacing or you name it, the difference to the user is minimal.  It is just the L written differently or the S moves closer to the T or it's very tiny  changes most often, but many pixels change because the follow up letters move to, if  the S moves closer to the T.  So this is in 12.3 screenshot.  And if I flip the fourth and back, you will ignore the left side, but only concentrate  on the license part.  You might spot the difference if I do this, but if I asked you which is which, it's already  getting complicated.  But to OpenQA, does it difference?  Here we have the same exercise again on the left side before, right side after.  And now I was not really sure how it is visible, but the pixel difference at the bottom contains  quite some noise.  Is this visible?  I see nodding.  So this is a 34% match.  This is how it looks like on OpenQA internally.  Okay.  The S-Hertz green is for gruy testing or for installation tests, the most common call actually,  but it is important to understand that tests, the test code is blocking code.  So in your test, you do not care what happens to the system under test.  You're only writing the S-Hertz or the commands.  And in between the system under test and QAMU and the backend, all of them continue running,  but the tests actually block.  So you do an S-Hertz screen on X-Term.  This might take a while depending on what happened else or you do a S-Hertz screen system  booted and then you reboot and then a S-Hertz screen waiting for the booted desktop.  And this can take minutes, but in between this call will just block.  So after you have the assertion that the X-Term is there, you type some string.  You don't care how long this takes and what it does.  And in the end, you do a S-Hertz screen on Hello-Type, which means once the string in  the font is rendered in the font you're expecting, it will actually pass.  All these three commands are blocking calls.  So what happens in the background, if I jump back to the architecture that I should have  copied, easy to video, we'll get the command from the test runner that there's supposed  to be an assert screen.  But easy to video will pass some commands to the backend to prepare the reference screen  on the needles matching to it and then will continuously polling the backend if it found  something in the current screen shot.  And only when the backend says there's a timeout or there is a match found, it will pass back  to the test runner the result.  And in between, it will await commands from that what you see as commands there in case  the web UI decided to inject new needles or the testers had stopped on the web UI and  all this happens in the easy to video program.  This is what I named here as an event loop.  And the actual grabbing the screen from VNC, the actual check the needles database, check  the needle comparison that we just saw, all this happens in the backend process.  And the backend process is also the one that has to take care that we grab a video, a screenshot  regularly from the VNC and pipe it into the video so that we are later on can have a regularly  continuous video so that every frame also is a repeated number of time.  So I want actually the last thing I have is our consoles.  Cables are a very important concept to support other platforms.  When we had only QEMU, it was QEMU like simulations.  It was really easy to just have send key alt F2 to switch to another console or open an  X term or so we had all these send keys that are very Linux specific or QEMU specific even.  But now we have consoles that you can drive through the test API.  So you do select console, user console and depending on the platform you are on, you  will either switch to alt F2, Linux console or to some X term that has some SSH connection  open to the system under test or the other console would be root console or we have lock  console or we have tons of consoles.  And the QEMU consoles that we have are all so called TTY consoles.  They are bound to one specific Linux terminal.  But they are only second most used.  The one that is most prominent in QEMU test is called SUT and this is the one that actually  talks VNC to QEMU.  So this is the one that does the assert screen.  This is the one that will type string into the currently selected TTY console and so on.  And that's what brings us to the last point on the slide that these consoles can be stacked.  So we have a TTY console that is actually connected to the SUT console because if you  do type string it will type into the TTY console but actually use the SUT console to do this  over VNC.  And there are other examples where we have this X3 270 terminal that is running in some  local X VNC.  There's also a console and so on.  Really fast shit.  The example I just mentioned is where we have X terms running locally on the worker as part  of the backend process that will actually drive commands on another machine that you  can't see because it's somewhere remote and we don't have a video of it.  But a local X VNC makes it possible that we can actually talk VNC to that console as well.  So we can see what is typed locally in this terminal and then have a video of it and we  can assert screen but when we do assert screens on this local X term running the commands  on a remote machine it's more complicated.  That's why I put it at the end.  But the key thing is that the tests that you write are very, very similar to what you had  for QEMO.  So when we introduced this IPMI backend to test real hardware we had to change one test  that was actually booting the system and after that we could use the same installation tests  that we had before that were asset screening, send type and so on.  And when we introduced or when we worked on S390 support it was a bit more complicated  especially as we had to introduce the console handling at all.  But in the end most of the tests are shared between all platforms.  So this is some benefit that we have now.  We can test let's say almost every hardware with the same set of tests as long as we have  SSH connection to it and we can power cycle it.  This is the VNC part that I mentioned initially for having a common factor between the back  ends is lost then but we will use some local X term to have the screen but for we can boot  the machines remotely, we can install them remotely, we can test things on them remotely.  This is all things we can drive so we don't need a lot from the hardware to be tested  as long as we don't need to see that it's video.  But for servers this is the perfect match and this brings even more test capabilities  to the system.  Good.  This leaves us five minutes for answers.  Any questions for that?  I think someone has to press here.  Test, test.  I have a question regarding comparing needles.  You described the algorithm and my question is probably this task comparing two images  is solved by many different ways.  This algorithm is something which is developed inside the open QA or it's some reuse of some  existing algorithm.  And in both cases if you answer yes or no why you select this way of doing comparison?  I must admit I did not really understand the question partly because of the audio and partly  because of your pronunciation.  The question is the algorithm which you are using to comparing needles.  This algorithm is something generic used outside open QA2 or it's something developed  inside open QA for solving this problem?  The algorithm that we use to have to do this, this is basically the backbone of the whole  needle matching.  This is based on open CVs template matching.  If you Google template matching you will find other examples of people using this.  For example automated autonomous test cars driving.  They also do something very similar to detect traffic signs.  When you drive they have all these cameras in your cars and they will detect traffic  signs just as we are detecting needles.  A bit more complicated because they also have to adopt to having different viewpoints.  So if you are coming from this side of the traffic sign you still have to detect it then  if you come from this side.  But it's basically the same algorithms there.  Thank you.  Hi.  I must admit that I don't have very deep knowledge with open QA but wouldn't it be possible  to use something like OCR to detect fonts to circumvent the pixel matching?  Do you know what I mean?  The fact is we did some experiments with it.  We started with that and we continued with that.  But the fact is that OCR is just as unreliable as this thing is.  If you have to train the OCR engine for every font change you have the same problem there  than we have right now.  The advantage of OCR would be that for example during installation you have tons of text scrolling  through and there's no really good way to detect an error for example.  So a warning that you want to assert that there's no warning scrolling through.  And if you had OCR you could actually make this happen.  But the problem is as long as OCR is so unreliable as it is right now or most likely ever will  be if you ask me.  The chances that you will miss these warnings is just as slim as currently.  So that's why OCR sounds reasonable but it's not so well suited in practice.  For console tests we don't do it at all and for GUI testing we don't care about the text  so much.  So that OCR is a lot of investment and there is support for it.  But whenever someone finishes the support or bug fixes it to work again, he soon realizes  or she soon realizes it's not really worth it and then it puts it again in the code base.  So we use Tesseract for that which is already quite good OCR engine and I use it for private  projects where you can train on the font used.  For example in my case in a dictionary but for having text in a random screenshot it's  just not suited.  Thanks.  Okay.  There are no more questions.  I'm done after exactly half an hour.  Thank you for your patience.
openSUSE_Conference_2017_How_openQA_works_vGpH4SQ9rQo_0_203.png openSUSE_Conference_2017_How_openQA_works_vGpH4SQ9rQo_204_2231.png openSUSE_Conference_2017_How_openQA_works_vGpH4SQ9rQo_2232_18983.png openSUSE_Conference_2017_How_openQA_works_vGpH4SQ9rQo_18984_19547.png openSUSE_Conference_2017_How_openQA_works_vGpH4SQ9rQo_19548_21179.png openSUSE_Conference_2017_How_openQA_works_vGpH4SQ9rQo_21180_21227.png openSUSE_Conference_2017_How_openQA_works_vGpH4SQ9rQo_21228_24863.png openSUSE_Conference_2017_How_openQA_works_vGpH4SQ9rQo_24864_24911.png openSUSE_Conference_2017_How_openQA_works_vGpH4SQ9rQo_24912_26459.png openSUSE_Conference_2017_How_openQA_works_vGpH4SQ9rQo_26460_26555.png openSUSE_Conference_2017_How_openQA_works_vGpH4SQ9rQo_26556_28115.png openSUSE_Conference_2017_How_openQA_works_vGpH4SQ9rQo_28116_36131.png openSUSE_Conference_2017_How_openQA_works_vGpH4SQ9rQo_36132_38531.png openSUSE_Conference_2017_How_openQA_works_vGpH4SQ9rQo_38532_38579.png openSUSE_Conference_2017_How_openQA_works_vGpH4SQ9rQo_38580_39215.png openSUSE_Conference_2017_How_openQA_works_vGpH4SQ9rQo_39216_39743.png openSUSE_Conference_2017_How_openQA_works_vGpH4SQ9rQo_39744_39803.png openSUSE_Conference_2017_How_openQA_works_vGpH4SQ9rQo_39804_39851.png openSUSE_Conference_2017_How_openQA_works_vGpH4SQ9rQo_39852_39899.png openSUSE_Conference_2017_How_openQA_works_vGpH4SQ9rQo_39900_39995.png openSUSE_Conference_2017_How_openQA_works_vGpH4SQ9rQo_39996_40223.png openSUSE_Conference_2017_How_openQA_works_vGpH4SQ9rQo_40224_40271.png openSUSE_Conference_2017_How_openQA_works_vGpH4SQ9rQo_40272_40331.png openSUSE_Conference_2017_How_openQA_works_vGpH4SQ9rQo_40332_40799.png openSUSE_Conference_2017_How_openQA_works_vGpH4SQ9rQo_40800_41015.png openSUSE_Conference_2017_How_openQA_works_vGpH4SQ9rQo_41016_41099.png openSUSE_Conference_2017_How_openQA_works_vGpH4SQ9rQo_41100_41807.png openSUSE_Conference_2017_How_openQA_works_vGpH4SQ9rQo_41808_41987.png openSUSE_Conference_2017_How_openQA_works_vGpH4SQ9rQo_41988_42731.png openSUSE_Conference_2017_How_openQA_works_vGpH4SQ9rQo_42732_43535.png openSUSE_Conference_2017_How_openQA_works_vGpH4SQ9rQo_43536_43595.png openSUSE_Conference_2017_How_openQA_works_vGpH4SQ9rQo_43596_43655.png openSUSE_Conference_2017_How_openQA_works_vGpH4SQ9rQo_43656_43835.png openSUSE_Conference_2017_How_openQA_works_vGpH4SQ9rQo_43836_44219.png openSUSE_Conference_2017_How_openQA_works_vGpH4SQ9rQo_44220_44279.png openSUSE_Conference_2017_How_openQA_works_vGpH4SQ9rQo_44280_44387.png openSUSE_Conference_2017_How_openQA_works_vGpH4SQ9rQo_44388_44435.png openSUSE_Conference_2017_How_openQA_works_vGpH4SQ9rQo_44436_44563.png
Hi. This, my talk is about OpenQA and how it works in the background. My plan was to have this in the line of OpenQA talks that piled up from beginners to experts. I somehow ended up being the first talk at all in the conference. So let's see how this works out. About me, I'm Stefan Kullo. I worked for SUSE for a long time. I was the release manager of Open SUSE until the release of Leap42.1. And currently I'm acting as OpenQA architect and test automation champion or however you want to name it. In my private life, I have two kids that I'm trying to teach free software. No, I don't. This talk will be about things you don't yet, you might not yet know about OpenQA, the architecture on how the whole system works internally and implementation details that would, if you know them, make you a better test driver in OpenQA. So you see, this is already some, I require some knowledge about OpenQA. So I will not have a beginner's guide on how to write tests. And I will not, yeah, you can still stay here if you don't know shit about OpenQA. I tried to have some introduction, but I might lose you quickly. So let's see. So OpenQA started in 2010 by Bernhard Wiedemann, our camera kind, to create videos from Open SUSE installations. And the name, EZO2Video, still sticks from that time. End of 2011 was the first release of OpenQA. You can still find it in the archives of news OpenSuser.org. And, but back then it was still mainly a tool for Bernhard and some selected others to create better bug reports. And this is how it looked like. So it was, if you know OpenQA now, it is not so much different. And actually some of the features that are on the left are we are currently re-implementing to track closed bug reports. So this is how it currently looks like. So it's still about GNOME and KDE tests, if you see, if you compare. It's very hard to read, but trust me, it's about desktop tests. Oh, good. The architecture is nowadays mainly about one central web interface, which contains a classic web application that has HTTP and a REST interface for an API. And all users use this as an entrance point to the whole system. There's some scheduler. There's some scheduler that keeps the jobs rotated between the workers and there's a database. And all these three components communicate through debas within one VM that I labeled here, WebUI. And then you have what I tried to express on the right, several workers that connect to this service, to this service, and take jobs and run the OpenQA tests. One of these workers I zoomed in at the bottom. You have several instances of a process called either-to-video running. This one splits into three in between. We have one test runner that actually runs the Perl tests. You have one commands server that takes commands from the outside, including the system under test. And the WebUI, in case you're using the interactive mode of the WebUI. And then there is the actual backend, which can fork several other processes, including QAMU or Lipvert commands. Or in case of S390 support an X3270 term or IPMI commands. So the backends are actually the real... I really should slow down. I was worried that I would have too many slides, but the backends are the real troublemakers. They share some common properties. One of them is that they can talk VNC. One of them is that they are able to power management the system under test. In QAMU this is really easy because you can just start and stop QAMU. But if you're talking about an S390 guest or an IPMI controlled hardware metal, this is more complicated. And the third one is in our case that we want to have access to the serial console to be able to have better console tests. The main backend still use is QAMU using KVM. And this QAMU one can emulate a lot of hardware setups. We have, for example, we have one case where we emulate in specific Dell laptop. We can have UEFI BIOS, we have legacy BIOS, we have several IOPAS systems. We have IDE SCSI, we have NVMe lately. We can emulate a RAID system, software RAID, we can have multi-path and so on. We can all emulate this just with QAMU. Now let me continue talking about backends for a while. But we also have other backends that, for example, connect to an lib-vert domain or host through SSH. So it can be actually everything. We have Hyper-V support through a VNC hop implemented. We have some S390 already mentioned. And these are all invisible to the test driver because the tests do the very same thing. Unless you're very specific, for example, reboot, but all the other stuff is done through a very generic API. And one core component is the needle matching that I would like to dig into further. A needle is a region of interest in a reference screen. Most of you that are right in the stock will know that already. But to give you an example, this is actually the pop-up that happens when you press F6, I think, on an OpenZoozer installation. It gives you a way to edit the HTTP installation URL. But this is actually only a zoom in. And the web UI will actually gray out everything else. That is not of interest. So let's give you a more detailed example. This, for example, is an quite old KDE screenshot with an X term started. So this is, we have the test driver wanted to verify that the X term actually started. So we had a reference screen. And this reference screen contains a region of interest. And that is this region that is not grayed out here. This is the part that identifies this screen as the X term has finished starting. The rest is not of interest. So this is the needle. This is what we call the needle. So if I go now on another screen where you don't have KDE, but still there's an X term started, we are asking OpenQA to assert that there's an X term started by means of having a third screen X term. And OpenQA would look at this screen and the first thing it does is it grays out the screen to 16 shades of gray and blurs the whole thing. I'm not sure how visible the difference is from behind, but this is the clear picture and this is the blurred picture. I guess from behind it's not obvious, but it's... So basically OpenQA looks at the screen as a very old snake. It's not... Some people think that OpenQA compares pixels by pixels, but it's not even close to that. The next thing it will do is look at this great, great out blurred screen to identify the most likely spot that the needle is. So this is some brain dump of OpenQA, let's say. The wider the pixel is, the more likely it contains the region of interest. What is invisible to the human eye is that it's actually exactly there where we headed the highest value. And what we are doing with this information is we're going through the likely spots and evaluating the actual pixel diff of this 16 shades of gray. And the number of differences we see in this area is counted and weighted. So we are not taking every error as an error, but we look on how big the area was and how many differences we saw. So the larger the area, the more forgiving OpenQA is to differences. And then we have some, let's say, magic formula that decides how many percentages this correlates to. This formula has grown over the time because we changed it, we changed the algorithm after we had tons of needles. So we had to adopt the formula to look like the old system, but it was still, give out similar match numbers. But in this very X term example, so you have Bernhardt on the screen, always, actually. On the left you have the KDE screenshot, on the right you have the GNOME screenshot, only blurred, grayscaled, the region of interest. And the brain dump of OpenQA says it's completely black. So the difference is exactly zero at every pixel, so it's 100% match. Any questions so far? Good. So what I think that is inherited from here is that big changes are obviously problematic, but what is not so obvious is that small changes to layouts can create quite some noise in the pixel diffs. The one prominent example is if we change the fonts or the font weighting or the font spacing or you name it, the difference to the user is minimal. It is just the L written differently or the S moves closer to the T or it's very tiny changes most often, but many pixels change because the follow up letters move to, if the S moves closer to the T. So this is in 12.3 screenshot. And if I flip the fourth and back, you will ignore the left side, but only concentrate on the license part. You might spot the difference if I do this, but if I asked you which is which, it's already getting complicated. But to OpenQA, does it difference? Here we have the same exercise again on the left side before, right side after. And now I was not really sure how it is visible, but the pixel difference at the bottom contains quite some noise. Is this visible? I see nodding. So this is a 34% match. This is how it looks like on OpenQA internally. Okay. The S-Hertz green is for gruy testing or for installation tests, the most common call actually, but it is important to understand that tests, the test code is blocking code. So in your test, you do not care what happens to the system under test. You're only writing the S-Hertz or the commands. And in between the system under test and QAMU and the backend, all of them continue running, but the tests actually block. So you do an S-Hertz screen on X-Term. This might take a while depending on what happened else or you do a S-Hertz screen system booted and then you reboot and then a S-Hertz screen waiting for the booted desktop. And this can take minutes, but in between this call will just block. So after you have the assertion that the X-Term is there, you type some string. You don't care how long this takes and what it does. And in the end, you do a S-Hertz screen on Hello-Type, which means once the string in the font is rendered in the font you're expecting, it will actually pass. All these three commands are blocking calls. So what happens in the background, if I jump back to the architecture that I should have copied, easy to video, we'll get the command from the test runner that there's supposed to be an assert screen. But easy to video will pass some commands to the backend to prepare the reference screen on the needles matching to it and then will continuously polling the backend if it found something in the current screen shot. And only when the backend says there's a timeout or there is a match found, it will pass back to the test runner the result. And in between, it will await commands from that what you see as commands there in case the web UI decided to inject new needles or the testers had stopped on the web UI and all this happens in the easy to video program. This is what I named here as an event loop. And the actual grabbing the screen from VNC, the actual check the needles database, check the needle comparison that we just saw, all this happens in the backend process. And the backend process is also the one that has to take care that we grab a video, a screenshot regularly from the VNC and pipe it into the video so that we are later on can have a regularly continuous video so that every frame also is a repeated number of time. So I want actually the last thing I have is our consoles. Cables are a very important concept to support other platforms. When we had only QEMU, it was QEMU like simulations. It was really easy to just have send key alt F2 to switch to another console or open an X term or so we had all these send keys that are very Linux specific or QEMU specific even. But now we have consoles that you can drive through the test API. So you do select console, user console and depending on the platform you are on, you will either switch to alt F2, Linux console or to some X term that has some SSH connection open to the system under test or the other console would be root console or we have lock console or we have tons of consoles. And the QEMU consoles that we have are all so called TTY consoles. They are bound to one specific Linux terminal. But they are only second most used. The one that is most prominent in QEMU test is called SUT and this is the one that actually talks VNC to QEMU. So this is the one that does the assert screen. This is the one that will type string into the currently selected TTY console and so on. And that's what brings us to the last point on the slide that these consoles can be stacked. So we have a TTY console that is actually connected to the SUT console because if you do type string it will type into the TTY console but actually use the SUT console to do this over VNC. And there are other examples where we have this X3 270 terminal that is running in some local X VNC. There's also a console and so on. Really fast shit. The example I just mentioned is where we have X terms running locally on the worker as part of the backend process that will actually drive commands on another machine that you can't see because it's somewhere remote and we don't have a video of it. But a local X VNC makes it possible that we can actually talk VNC to that console as well. So we can see what is typed locally in this terminal and then have a video of it and we can assert screen but when we do assert screens on this local X term running the commands on a remote machine it's more complicated. That's why I put it at the end. But the key thing is that the tests that you write are very, very similar to what you had for QEMO. So when we introduced this IPMI backend to test real hardware we had to change one test that was actually booting the system and after that we could use the same installation tests that we had before that were asset screening, send type and so on. And when we introduced or when we worked on S390 support it was a bit more complicated especially as we had to introduce the console handling at all. But in the end most of the tests are shared between all platforms. So this is some benefit that we have now. We can test let's say almost every hardware with the same set of tests as long as we have SSH connection to it and we can power cycle it. This is the VNC part that I mentioned initially for having a common factor between the back ends is lost then but we will use some local X term to have the screen but for we can boot the machines remotely, we can install them remotely, we can test things on them remotely. This is all things we can drive so we don't need a lot from the hardware to be tested as long as we don't need to see that it's video. But for servers this is the perfect match and this brings even more test capabilities to the system. Good. This leaves us five minutes for answers. Any questions for that? I think someone has to press here. Test, test. I have a question regarding comparing needles. You described the algorithm and my question is probably this task comparing two images is solved by many different ways. This algorithm is something which is developed inside the open QA or it's some reuse of some existing algorithm. And in both cases if you answer yes or no why you select this way of doing comparison? I must admit I did not really understand the question partly because of the audio and partly because of your pronunciation. The question is the algorithm which you are using to comparing needles. This algorithm is something generic used outside open QA2 or it's something developed inside open QA for solving this problem? The algorithm that we use to have to do this, this is basically the backbone of the whole needle matching. This is based on open CVs template matching. If you Google template matching you will find other examples of people using this. For example automated autonomous test cars driving. They also do something very similar to detect traffic signs. When you drive they have all these cameras in your cars and they will detect traffic signs just as we are detecting needles. A bit more complicated because they also have to adopt to having different viewpoints. So if you are coming from this side of the traffic sign you still have to detect it then if you come from this side. But it's basically the same algorithms there. Thank you. Hi. I must admit that I don't have very deep knowledge with open QA but wouldn't it be possible to use something like OCR to detect fonts to circumvent the pixel matching? Do you know what I mean? The fact is we did some experiments with it. We started with that and we continued with that. But the fact is that OCR is just as unreliable as this thing is. If you have to train the OCR engine for every font change you have the same problem there than we have right now. The advantage of OCR would be that for example during installation you have tons of text scrolling through and there's no really good way to detect an error for example. So a warning that you want to assert that there's no warning scrolling through. And if you had OCR you could actually make this happen. But the problem is as long as OCR is so unreliable as it is right now or most likely ever will be if you ask me. The chances that you will miss these warnings is just as slim as currently. So that's why OCR sounds reasonable but it's not so well suited in practice. For console tests we don't do it at all and for GUI testing we don't care about the text so much. So that OCR is a lot of investment and there is support for it. But whenever someone finishes the support or bug fixes it to work again, he soon realizes or she soon realizes it's not really worth it and then it puts it again in the code base. So we use Tesseract for that which is already quite good OCR engine and I use it for private projects where you can train on the font used. For example in my case in a dictionary but for having text in a random screenshot it's just not suited. Thanks. Okay. There are no more questions. I'm done after exactly half an hour. Thank you for your patience.
too many slides, but the backends are the real it's 100% match. Any questions so far? Good. So long this takes and what it does. And in the end, can drive through the test API. So you do select locally on the worker as part of the backend