Mutants, tests and zombies
Todorov, Alexander
Mutation testing is a technique in which the software under test is modifiedin a controlled manner to produce a mutant. Then test cases are executedagainst each mutant. This helps answer the question "How good is our testsuite?". For example if we have made the following change: - if A and B: + if A or B: and our test suite reports a PASS result that means we are not doing a goodjob at detecting possible errors. I have been using mutation testing for production grade software in bothPython and Ruby and I'm also the most active contributor to Cosmic Ray, themutation testing tool for Python. In this talk I will explain how mutationtesting works and what it can be used for. I will give practical examples ofcode which wasn't tested and how to test it and also examples of bugs thatI've found. I will also mention some differences between the Python and Rubytools.
 Hello everyone. So my name is Alex. I have been testing open source software for the last 10 years. This is how to reach me if you like. And this is how I look. No pretty close. So today I'm going to talk about mutants and zombies but not the ones from the movie. It's slightly different. And I have some code examples. So first let me get an idea how many of you are familiar with Python at least to some degree. Okay, good. And how many of you are familiar with Ruby? Okay, great. So don't worry. The examples are not very hard to understand. So let's take this piece of code. It's a simple class describing one model in the Rails application. There's only one field that goes into the database. And this is only one method which is actually a really stupid method just returns an uppercase string of that field. And as you can see from this screenshot, this class has 100% code coverage. And the question that I'm asking myself as a tester and the question that I expect everybody should be asking themselves here is, is my test sweet good enough? And good enough in terms whenever there's some change in the software under test, is my test suitable to detect if this change will break something or the change goes undetected and possibly goes into production and something bad really happens? And mutation testing can help you answer this question. But first I need to explain what a mutation is. So a mutation is usually a very small change in the software which somehow changes the behavior of the software. Mutations can come from comparison operator and if statements. And if we take the example on the screen, the mutation is replacing the great done operator with a less than operator. And if you apply this mutation to your code and read it, it becomes if h is less than 18, then by beer. Now imagine that you are an online store and suddenly you start doing this, sell beer to small children and this is not good. Then another possible source of mutations is constant values. You can replace these values with something else. And in this example, we replace truth false and because it's very early in the morning yet, if we didn't have our coffees, we would be looking like this. Another possible source of mutations is loops. We may, for example, modify the loop condition or in this example, we may change break to continue and we get an endless loop. And when you have a endless loop, this is what happens. So this is again an example from production and all my examples are taken from production environments today. Well, not the beer one, but by definition, mutation testing is a very small thing. Mutation testing is a technique and also the tools for mutation testing. They modify your software under test. They don't modify your test suite. You keep the test suite separate and for each mutant that is produced in your software under test, you execute the test suite again and again and again. And this gives you a pretty good idea at the end how good your test suite is. It can tell you places where you have tests, but they don't do a very good job of doing finding all possible things that may change. Or they can also tell you places where you have missing tests, but you already notice because we use coverage. So mutation testing is really good for telling you where you need to make better tests. And the idea behind this is some of the mutations which you see and which the tools support try to mimic errors which developers may do while writing code, for example, plus one, minus one errors, delete something by accident and just commit this to source control, stuff like that. And other mutations which you may see, they are purely artificial, but somehow they help us validate the test conditions, the test environment which we run our test suite in and help us expose something that's missing. The algorithm for mutation testing is actually very simple. So we run three loops, one after the other. First one is we go through the all mutation operators that our tool in particular can support, these are stuff which the tool knows how to change with something else, regardless whatever that else may be. Then for each operator we find the places in the source code where that operator is used and replace it with something else. Most operators and conditions can lead to only one another type of mutation, but sometimes you can produce different mutations for only one place in the code, like with the comparison operators. And then of course you execute the test suite. There are three simple rules to kill mutants which you must remember. First when you execute the test suite against the non-modified version of the program, everything should pass and this is a hard requirement, you cannot go without it. If something fails obviously your software doesn't work or your test suite is broken and you need to take measures. And if you have flake tests and sometimes they pass, sometimes they fail and you have no idea why this is, obviously your mutation results will also be unreliable and you need to take measures to fix your flake tests. The second rule is when you run the test suite against a mutant that is a modified version of your program, you expect the result to be failed and that's a good thing. When the test suite fails then we say that the mutant was killed or the mutant died and this means we had at least one assertion or one condition in the test suite which wasn't met and the test suite failed. So that means at least we have one test which is able to detect this change and tell us the software was modified. And the last thing is which you don't want to happen when you run your test suite against the mutant and the result is pass. This is a bad thing. Then we say we have a zombie or that the mutant survived and as you know from the movie zombies are dating that go around and try to eat you and now imagine you make some change in the code, run the test suite it passes but it doesn't really understand there's anything change and now this change goes into production and suddenly this becomes a bug and tries to eat you. So these are the three rules. Testing against the non-modified version should pass always. Testing against a mutant should fail. This is the way mutation test tools figure out when you kill the mutant. When something fails in the test suite the tool says you kill the mutant. Okay now. Let's play a little game. I am going to show you possible mutations and because you all know about testing you are going to tell me what test cases I need to write to kill the mutants. I am using the code from earlier the one with the 100% coverage and only take this method. See this is a string variable and we return the uppercase of the string variable. That's what it does. So very simple. First possible mutation is instead of returning the uppercase string my method under test returns new value. So what test can you propose so I can kill this? Yep. Yes, obviously not new. Yes, correct. So we execute the method under test and expect the result to obviously not be new. If we apply the mutation and run the test the method will return new and the expectation will fail. Everything will fail and we kill the mutant. Next possible mutation is instead of returning the uppercase string I am returning self because this method is part of the class and I have access to the self object so I can do this. What test do I need here to kill this mutation? OK, type string, correct. So I must be checking what the type of the return value is and if I expect only string then it must be only string and nothing else. Another possible mutation is instead of returning the uppercase I just return the value of this variable as is. So how do I kill this? OK. So I start with a value which is in a lower case and then I execute the method under test and expect this to be in a uppercase. And I use this with just a constant because it's easier for me. No. So this one for example will kill the first and the second zombies. And here if I start with new, OK, let me think. With the new test. Yeah it might kill also the first one. So sometimes yeah one of the test is enough to kill more mutants. But actually this is the way I discovered them and I developed the test for them and actually didn't go back to think whether or not I have some test which is not needed. And the last example from the game is replacing this ampersand. This is the new safety navigation operator in Ruby and I just replace it with a dot. So for the folks who don't work with Ruby, the new safety operator works this way. If the object on the left is new, then the result of this operator is new and nothing else is executed. And if the object on the left of the operator is not new, the execution continues to the right. And when we replace this with a dot, then if a language code is new, we just get a runtime exception because a new object doesn't have another method called uppercase. What's the difference? So we do this mutation and what test do I need to kill this? Should what? Should not throw. Should not throw. Okay, how do you assert on that? Okay, well at least I don't know how can I assert that there was no exception but I can set this variable to new, execute the metro under test and if there is an exception, the test will fail anyway so I don't have to assert there wasn't an exception but I can assert that the result was new. So if the framework asserts whether an exception was not raised, then of course that's a valid answer but I don't know how to do this with Ruby. Okay. It collides if you read the examples as is on the slide. What I haven't shown is that this variable has a non-new value by default. So that works then. But yeah, good catch. Now let's try and find some bugs with mutation testing. One bug that I was able to discover just by using mutation testing in a project, not actively looking for the bug is this. So we have a class called network which represents the networking settings on your Linux computer and there's an attribute called self-device which is the name of the network interface and this method for equality is obviously wrong, as highlighted here and we'll see why. So if self-device is not on an empty string, that piece of the Boolean expression is always evaluated to false and the entire return value is always false. So it doesn't matter what's the other object you're comparing to, this method always returns false and if self-device isn't non or it's not an empty string, then which means it has some meaningful value in the software, this is evaluated to true and the Boolean expression always depends on the second part. So and that's actually the fix, just remove this first part of the Boolean expression and here we go. The reason this statement detected for I guess about seven or eight years is there wasn't a single test in the test suite which tried to test this equality method when some of the attributes was an empty string or was none. So they were always testing with some valid values and this went undetected for many, many years. Now also in the software under test, when you see normal conditions, this value is always, it always has some value so it doesn't return false or something bad happens. Another book which I was able to find is as shown, I have two classes, the second class inherits from the first one and you see both classes have parameter code speed limit in the init method and both parameters have default values. Now this is perfectly valid Python code so there's nothing wrong with the way it's written. I run this through mutation testing and I get a surviving mutant. The reason here is this is a constant change, the two for Python adds plus one to any integer constant and to see what happens. And I immediately know the reason for this is that I don't have any test which creates an object from this class and asserts what the default value of this attribute should be. So I create my test like this, just create an object from the class under test and assert what the default value should be and the test immediately fails of course. And the reason for this is if you look closely, you see that this parameter, I'm not sending that to the init method of the parent class so it must go here after self. And when I'm not doing this, the parent class has its default value so it uses 50 instead of 90 and that's why my test fails. And this is also very subtle, it stayed undetected for many, many years and also the reason for not being detected is the attribute to this value is assigned to, is never used in the software under test. It was meant to be used by external clients of this software, the software under test in this case is a library so it's used by other tools and apparently nobody bothered to check whether or not the default value is it should be. Another possible book very close to the previous one is again I have two classes, the second class inherits from the first one. Again I have a parameter with the default value and this time notice I am actually sending this to the init method of the parent class and everything must be fine. I run this through my mutation testing tool, I get the same surviving mutants so again I write the same type of test, create an object and assert what the default value of the attribute should be. And this time I get another type of exception, it's an attribute error telling me object from the Mock Pro 8 class does not have an attribute called speed limit. And at this point I start wondering why this is, so I look here, everything looks cool, I look here, everything looks okay and I traverse back to the parent class and immediately here I notice first there's no parameter called speed limit and then something starts to look not right. And then I look in the body of the parent init method and nobody cares about whether or not there's a parameter called speed limit, nobody sets an attribute called self speed limit. So that's the problem, you know, I don't have this. One possible fix is to, if you really need this to just take care of it in the class under test and set the attribute or another possible fix is to delete everything related to this parameter, I don't bother with it and that was actually the fix in production. Another thing which mutation testing is really good at is forcing you to look at your source code and refactor it and places where mutation testing really shines is where you have if statements and comparisons and lots of Boolean expressions, stuff like that. The reason for this is we have many mutations in places like this. So in this example on the screen we have around 100 different mutations. Every comparison operator can lead to almost 10 different mutations. So equals here can be replaced with non-equals with less than or greater than equals in Python is not and also in and not in operators, so that many. Boolean n can be replaced with Boolean or we can negate the entire Boolean expressions. Also in the tool for Ruby you can replace the Boolean expression with a true or false constant and I think also you can change only parts of it so you can replace this with true or false and then leave the rest as is. And the Python 2 doesn't do this at the moment but it's fairly easy to add. So this goes through mutation testing and something, you know, there are surviving mutants and when you start looking at it you notice the pattern highlighted in red. So my thought is I can delete this and move the second block of fifth statements to the right and it becomes a little bit more clear. And then I notice another pattern that whatever value is, I'm looking for an attribute under self-handle with the same name and do something with it. So I can refine this even further and use the get attribute function and this becomes like this. In reality this fits only on four lines instead of ten lines and it's much more easier to test and much more, you know, easier to read actually. And that's a good thing. So we've seen, you know, what mutation testing can do. It forces you to write better asserts and in my opinion when you have a complex software under test we not only should assert what the return values of our methods are but we also should assert what intermediate state or side effects are produced by the functions under test. And we all agree that it doesn't matter how much we try to write clean software, we always have these methods which do more than one thing at a time. So they do something, some calculations, return some value and oh by the way I've just set this attribute on the site just so you know. And that's what mutation testing is really helpful with. It helps you discover these things which you are not testing by mutating them so you can write your tests better. We saw, we can find some interesting bugs and we saw that we can find places which we can refactor. And the question still stands, is my test suite good enough? And another question, another side of this question is how do I measure how good my test suite is? Which metric do I use to tell whether or not my test suite is good? And you know, metrics is fairly controversial topic. I just want to mention some research that's been going on in the last few years. In 2015 at GTAC there was one lightning talk which says you know coverage is not a good metric because it doesn't give you a lot of information so go for mutation score, you know use mutation testing and measure how much of the mutants are killed. If you kill 100% of the mutants then you're good. And then last year at GTAC there was another researcher who basically said well you know the guys from last year they didn't do really good research, they didn't research on many software so we did better research and we claimed that the coverage metric like lying coverage and branch coverage is still the best metric in practice. They say the problem with mutation testing is it is very expensive to compute and in their research it gave only additional 4% of information which the guys didn't know already compared to coverage so they say you know use coverage, don't use mutation testing. And I decided to do a small experiment and see which one of these researchers is right. So my software under test is called Pelikan AB and this is a very small library, it's a plugin for Pelikan. Pelikan is a static HTML generator for Python which you can use to run your block or your source, siton. And Pelikan AB gives you one additional TAC for the templating markup which you can use to encode a varying layout for your website. So for example you can change the cover of links or covers of buttons stuff like that. The way to use this software is to define the AB experiment variable in the shell and run the make command. If you run make GitHub then the site gets rendered, everything is published directly to GitHub. The way to render several versions of your website is first start with the control version and then name each experiment by name and run make in a sequence like shown on the screen. And if you do this, the three commands then you get the control version of your website on experiment which is called 123 and everything about this experiment goes into a directory with the same name, the URL structure is updated and you can point your users to only to that experiment and see how they respond and stuff like that. And in the version under test we have 100% branch and line coverage of this software and we also have a book. There is a setting called delete output directory which is set to true by default and this setting isn't something the software under test controls. This lives into an external file where your website configuration goes. So stuff like your website name, your GitHub handle goes into that file as well. The bug is when that setting is set to true Pelikan will go and delete the output directory and delete all the HTML files and then start rendering them in a clean directory. So the result of this command sequence with the setting set to true is that you have deleted your entire website and only left with the last one. And imagine now you delete everything and type make GitHub and everything goes live. So that's a pretty good way to destroy your website. So I didn't have a test which will fail if this setting is set to true. So I decided to integrate mutation testing into the project and I wrote a few more tests to achieve 100% mutation coverage, mind you this is a very small library, very small plugin and the bug was still present. So this means I don't have any test which fails when this setting is set to true. And I started thinking, okay why I have these many tests and this bug is still present and the answer is you cannot discover this type of bug without looking at the external environment and that's why we need integration tests. So I added an integration test which simulates the external environment with the settings and simulates the make command then tries to verify what content has been rendered and whether or not it's correct and the test immediately failed of course. So then I said I fixed the bug but in reality just changed this setting to false and also added to check whether or not this is set to true just raise an exception in my software. And now we have 100% mutation coverage, 100% branch coverage and also we have at least one integration test and I'm thinking okay I must be good then you know if I have so many tests and I'm using these techniques then possibly my software is bug free and of course this is not true. I've added a pilot to the project and pilot was immediately able to discover this bug. This is a problem with how we code the super method instead of passing the class name. I was using a shortcut which is self.underscore class and this works perfectly fine when you use the software in its intended environment because we have only one class and self class is evaluated to that particular class's name and everything's fine. This becomes a problem the moment you try to inherit from the class under test and create a new class and do something different with it. Then when you code the class under test init method Python goes into a loop between the parent and the inherited class init methods. So if you want to learn more just head on to this pilot pull request number and there's a very detailed description why this is a problem and I am guilty of using this shortcut in Python but I've seen this in many many projects online at GitHub and I've seen this in popular software which is used by many people. So this tells me people don't have very good understanding of what self class actually means and how it works and that's why we keep using it in the wrong way. So to conclude the mutation versus coverage topic this is a link to my blog describing my experiment with more details and there are links to actual git commits so you can see what actually what code was changed and how it was changed. So I think when we first start doing testing what we look at is how much coverage we can do so we strive to write more tests to test as much as possible of the software under test and that's a good thing until some time when we go to 100% coverage we don't get any more information out of this metric so it doesn't do any good. Then we start looking at mutation testing and mutation testing tells us okay now you have some coverage and you are testing some stuff but then you know there's a lot more you can test and this is you know what you need to do and we start doing mutation testing and we get to 100% mutation coverage and when we get to 100% we don't get any more useful information out of that and we also need integration tests because of the external environment as you see from the examples we have different types of environments. One type of environment is the regular environment which our users will be using and another possible type of environment is a developer just taking our software and trying to do something else with it or you know build on top of it and we have no idea what these environments will be and how people will want to use our software so that's why we need different types of tests and you know possibly we also use different types of tools to detect the problems and deal with it and I do need more examples on the topic I will be continuing to explore this topic throughout the year but if you have examples which you can share with me or you know publish something to GitHub please send me an email. Now I will go to something more practical, speed of execution, so as you can imagine mutation testing is very slow and just to measure how slow I have taken a real world project from Fedora it's called Pi Kickstart, it is a text parser library used by the Fedora installation program, it's a medium-sized project with a little over 100 files I think all of these files are Python modules, they don't have many dependencies between each of them so that's good. Each module you know does some checking, a few if statements, there are hardly any loops in the code so you know very easy to understand actually, there is some text then they write some text as well as an output and this is a library which is meant to be used by other programs so it doesn't really do anything useful on its own. The project has a fairly good test suite over 90% coverage with a lot of tests and the other good thing about it is the files under the source directory, MapoMOS 1 to 1 with the files under the test directory so they have the same names. And first thing I did was take Cosmic Ray, the mutation testing tool for Python and told it, okay so here's the source directory which means load everything into memory, you can find under this directory in terms of modules, produce all the possible mutations you can produce on these modules and then here is the test directory, use the test runner to discover all possible cases you can discover and just start running and I let this run on my computer that took over six days. Then I became smarter and wrote a small script to go through the source directory, take only one file, load this into Cosmic Ray which means produce mutations but only for that module, don't bother about the rest of the software, only that module and oh by the way here is only one file in the test directory which contains your test so use only that for testing and not everything else. And that was faster, I've also added an option called fail fast which means whenever one of the tests fails we know that there was a failure and we'll kill the mutants so don't bother to execute the rest of the cases, this is an option for the test runner and I did some refactorings like stuff like if length of string is greater than zero, I've replaced this with only if string and this helped me save about 1000 mutations which were very obvious things and I let this run, the execution time was now a little over six hours and this is 20 times more improvement in speed of execution but still quite slow for any practical purposes I must say. So the way to use mutation testing at the moment in my opinion is if you're testing a very small library or a very small project then you can go full on into CI and just tell okay this is my common line to schedule my mutation testing jobs and let this run for 10, 15, 20 minutes and that should be fine. If your project is anything bigger than 200 lines this is not going to work very well but what you can do is create a commit hook or pull request hook which examines the payload and you can first thing you can do is schedule mutation testing only against the files which have been modified but by that particular commit or pull request and that should be faster. Next thing you can do depending on the tooling is go down from the module level to the class name and to the method name which has been modified. The Python 2 doesn't know anything about classes and modules it only knows, it doesn't know anything about classes and methods sorry, it only knows about modules so regardless if you have one class or 1000 classes in one module Python loads the entire module and starts doing mutations against everything. On the other hand the Ruby 2 knows about classes and methods I think and you can tell you know test only that particular stuff and you can try and go even further because this is a pull request or commit hook you have access to the actual diff and you can apply this to the source under test and assuming you've tested everything before and it's fine then you can schedule mutations only against the lines which were changed. So you can do this and it's not really impractical to do this. Another thing you can do is go into parallel again depending on the tools the Python 2 is built around salary so you can very easily hook this to some messaging back end like RabbitMQ and schedule hundreds and thousands of messages and let your infrastructure deploy Docker instances or virtual machines in the cloud and just run your tests in parallel and just get back the results and you can do this if you have a lot of money of course. Here is a list of some mutation testing tools. I have been using only the first two I mostly use Python and I contribute to Cosmic Ray for Python and I've used Mutant for Ruby but not very actively at the moment. The names to the rights are GitHub repositories. So as far as I know the tools on the top are based on the abstract syntax tree of the language and they are language specific so whatever you want to do with the tool it works on ASTs and modifies the nodes of the AST. One thing I like to do is look at the other tools and see what they're doing especially in terms of mutation operators and try to bring this to Python. So if you want to actively use mutation testing I really advise you to look at tools for other languages and see how they work and how they are doing because many of the tools are very new they are not very mature and on the bottom this is another tool for mutation testing called the Mule project which Alex Denisoff will be presenting later today so I'll definitely be checking this out. This is an LLVM base tool so it should work for several different languages. If you are into mutation testing also check this out. And now before I can go further I can take some questions from the audience if you like. Okay go. So the question is if we have a zombie but we have 100% code coverage does it mean that coverage was wrong? Yeah. Yeah okay so it doesn't mean coverage was wrong like I mean it doesn't mean the metric the measurement was wrong it was probably correct. There are other problems with coverage like for example if you have one line with the lots of with a long Boolean expression then it still gets covered regardless of how much of that expression is evaluated so you may be evaluating only the first part of the expression and still cover that line but you know the next 10 parts of the Boolean expression are not evaluated. Yeah well the way to count you know these cases you should take this up to the authors of tools who do coverage but there are many many publications online with respect to problems in coverage and why it's not really a good metric why you shouldn't use coverage and really for me coverage is a vanity metric it really tells you how much of the you know of the lines you've covered but nothing more it doesn't tell you anything more and if you have a zombie on a line which was covered this simply means you might have executed this line but probably you didn't assert on some condition or you know you assert it on one condition and you need it to assert on two conditions. I have a comment to that basically this is a difference between line coverage, statement coverage and branch coverage. Yes yes. 100% branch coverage then it's unlikely that you will have 100% coverage and have zombies but most of the tools count only line coverage and very few counties on statement coverage of the mentioned. We have a question there. Yes so this works really well with primitive operations that might change you know less than data that plus into plus two. What if you're not using our pre-tips in business that you know if you're not using library especially in JavaScript but I don't know if you're using those or your condition is scoring with functions. So the question is what if we don't use primitives, we use some library for the business logic how does mutation testing come into play. This is very dependent on the tools that you use. So for example the Python tool until recently it was very poor on mutation operators because it's a new tool and not many people use it. On the other hand the Ruby tool called Mutant I think this is one of the best tools currently in existence and it has tons of mutation operators, tons of conditions. It understands because it's used by people who work on commercial software they get paid to actually write test suites for commercial software and they support the tool for that reason. About JavaScript I don't really have no idea, I don't use JavaScript but really the take here is you need to know your tools very well and you need to know the software very well and that's when you can decide whether or not that's going to be useful. Maybe in your case you might write a plug-in or an extension to the tool and produce the mutation based on the functions in that particular library. Question here? So thank you for your talk. Academic claims that the biggest problem of mutation testing is equivalent mutants. Can you say something about them, how often do you see them in production or is it something that you struggle with and practice? So the question was about equivalent mutants. Equivalent mutants are things which change the code but don't really change behavior in any practical way. So for example we may have a less-done operator which in practice is equivalent to less-done or equal depending on the values we accept in the application. I don't have any concrete measure about this in practice but I think about 10 or 15 percent of the time I see equivalent mutants. Equivalent mutants because the behavior doesn't change, the syntax changes but the behavior doesn't change, the test suite doesn't fail and we cannot kill them and they just stay as is. In the project I use mutation testing for, I usually have some threshold like about 10 percent or something like that. And if the mutants score is above a certain line then I say we're good and then the CI system goes green and if you drop under that line then we go red and inspect what's happening and that threshold is based on how many equivalent mutants I have. So I try to get some idea about that and then go from there. I've talked to other people who do mutation testing and have been doing mutation testing for a long, long time. And this is a problem but still the benefits you get from asserting on all those different conditions and looking at your code base and understanding the software much more better which is the result of mutation testing, it just, you know, the benefit is greater than having to deal with equivalent mutants. Thank you. Okay. More questions or no questions? We have two minutes so if there are no more questions I'll just show you this very quickly. I have started to document my findings about mutation testing so first of all I don't forget them and some of them make really good examples. This is available on Read the Docs and it's also available on GitHub if you'd like to contribute. And it, since the last week we have also Chinese translation for this and I'll skip the trivial examples and show you the last one. So we have this in Python it's fairly common to have methods for equality which compare two objects by comparing that all of the attributes are equal. So in this example we have a class called sandwich and you can modify the meat of your sandwich and the bread type of your sandwich and we say that two sandwiches are equal if meat and bread attributes are equal and that's about it. You know, we have a safety check here if the other object is none, the editor falls and the not equals method is just a negation of the equals. And the way to test this with mutation testing is like so. So we create two test objects, sandwich one and sandwich two. By default all of their attributes have a value of empty string which is not shown on the screen. And first thing we do is test the test for equality they should be equal because everything is an empty string inside these objects and we also assert that the two objects are not, you know, when compared for non-equality it returns false and that takes care of half of the testing. Then we test the safety check compared to none and expect to not be equal and then comes the fun part. The way to, so this starts changing into different comparison operators and the way to test is like this. So we need to modify only one of the attributes at one of the test objects and leave the rest of the attributes and the other object unchanged and we need to do this for every single attribute. So when all the attributes are of the same type you just do this and if they're not of the same type you can do grouping or something like that to, you know, to modify this block. That's when the time is up so thank you very much.
 Hello everyone.  So my name is Alex.  I have been testing open source software for the last 10 years.  This is how to reach me if you like.  And this is how I look.  No pretty close.  So today I'm going to talk about mutants and zombies but not the ones from the movie.  It's slightly different.  And I have some code examples.  So first let me get an idea how many of you are familiar with Python at least to some  degree.  Okay, good.  And how many of you are familiar with Ruby?  Okay, great.  So don't worry.  The examples are not very hard to understand.  So let's take this piece of code.  It's a simple class describing one model in the Rails application.  There's only one field that goes into the database.  And this is only one method which is actually a really stupid method just returns an uppercase  string of that field.  And as you can see from this screenshot, this class has 100% code coverage.  And the question that I'm asking myself as a tester and the question that I expect everybody  should be asking themselves here is, is my test sweet good enough?  And good enough in terms whenever there's some change in the software under test, is  my test suitable to detect if this change will break something or the change goes undetected  and possibly goes into production and something bad really happens?  And mutation testing can help you answer this question.  But first I need to explain what a mutation is.  So a mutation is usually a very small change in the software which somehow changes the  behavior of the software.  Mutations can come from comparison operator and if statements.  And if we take the example on the screen, the mutation is replacing the great done operator  with a less than operator.  And if you apply this mutation to your code and read it, it becomes if h is less than  18, then by beer.  Now imagine that you are an online store and suddenly you start doing this, sell beer to  small children and this is not good.  Then another possible source of mutations is constant values.  You can replace these values with something else.  And in this example, we replace truth false and because it's very early in the morning  yet, if we didn't have our coffees, we would be looking like this.  Another possible source of mutations is loops.  We may, for example, modify the loop condition or in this example, we may change break to  continue and we get an endless loop.  And when you have a endless loop, this is what happens.  So this is again an example from production and all my examples are taken from production  environments today.  Well, not the beer one, but by definition, mutation testing is a very small thing.  Mutation testing is a technique and also the tools for mutation testing.  They modify your software under test.  They don't modify your test suite.  You keep the test suite separate and for each mutant that is produced in your software under  test, you execute the test suite again and again and again.  And this gives you a pretty good idea at the end how good your test suite is.  It can tell you places where you have tests, but they don't do a very good job of doing  finding all possible things that may change.  Or they can also tell you places where you have missing tests, but you already notice  because we use coverage.  So mutation testing is really good for telling you where you need to make better tests.  And the idea behind this is some of the mutations which you see and which the tools support  try to mimic errors which developers may do while writing code, for example, plus one,  minus one errors, delete something by accident and just commit this to source control, stuff  like that.  And other mutations which you may see, they are purely artificial, but somehow they help  us validate the test conditions, the test environment which we run our test suite in  and help us expose something that's missing.  The algorithm for mutation testing is actually very simple.  So we run three loops, one after the other.  First one is we go through the all mutation operators that our tool in particular can  support, these are stuff which the tool knows how to change with something else, regardless  whatever that else may be.  Then for each operator we find the places in the source code where that operator is  used and replace it with something else.  Most operators and conditions can lead to only one another type of mutation, but sometimes  you can produce different mutations for only one place in the code, like with the comparison  operators.  And then of course you execute the test suite.  There are three simple rules to kill mutants which you must remember.  First when you execute the test suite against the non-modified version of the program, everything  should pass and this is a hard requirement, you cannot go without it.  If something fails obviously your software doesn't work or your test suite is broken  and you need to take measures.  And if you have flake tests and sometimes they pass, sometimes they fail and you have  no idea why this is, obviously your mutation results will also be unreliable and you need  to take measures to fix your flake tests.  The second rule is when you run the test suite against a mutant that is a modified version  of your program, you expect the result to be failed and that's a good thing.  When the test suite fails then we say that the mutant was killed or the mutant died and  this means we had at least one assertion or one condition in the test suite which wasn't  met and the test suite failed.  So that means at least we have one test which is able to detect this change and tell us  the software was modified.  And the last thing is which you don't want to happen when you run your test suite against  the mutant and the result is pass.  This is a bad thing.  Then we say we have a zombie or that the mutant survived and as you know from the movie zombies  are dating that go around and try to eat you and now imagine you make some change in the  code, run the test suite it passes but it doesn't really understand there's anything  change and now this change goes into production and suddenly this becomes a bug and tries  to eat you.  So these are the three rules.  Testing against the non-modified version should pass always.  Testing against a mutant should fail.  This is the way mutation test tools figure out when you kill the mutant.  When something fails in the test suite the tool says you kill the mutant.  Okay now.  Let's play a little game.  I am going to show you possible mutations and because you all know about testing you  are going to tell me what test cases I need to write to kill the mutants.  I am using the code from earlier the one with the 100% coverage and only take this method.  See this is a string variable and we return the uppercase of the string variable.  That's what it does.  So very simple.  First possible mutation is instead of returning the uppercase string my method under test  returns new value.  So what test can you propose so I can kill this?  Yep.  Yes, obviously not new.  Yes, correct.  So we execute the method under test and expect the result to obviously not be new.  If we apply the mutation and run the test the method will return new and the expectation  will fail.  Everything will fail and we kill the mutant.  Next possible mutation is instead of returning the uppercase string I am returning self because  this method is part of the class and I have access to the self object so I can do this.  What test do I need here to kill this mutation?  OK, type string, correct.  So I must be checking what the type of the return value is and if I expect only string  then it must be only string and nothing else.  Another possible mutation is instead of returning the uppercase I just return the value of this  variable as is.  So how do I kill this?  OK.  So I start with a value which is in a lower case and then I execute the method under test  and expect this to be in a uppercase.  And I use this with just a constant because it's easier for me.  No.  So this one for example will kill the first and the second zombies.  And here if I start with new, OK, let me think.  With the new test.  Yeah it might kill also the first one.  So sometimes yeah one of the test is enough to kill more mutants.  But actually this is the way I discovered them and I developed the test for them and  actually didn't go back to think whether or not I have some test which is not needed.  And the last example from the game is replacing this ampersand.  This is the new safety navigation operator in Ruby and I just replace it with a dot.  So for the folks who don't work with Ruby, the new safety operator works this way.  If the object on the left is new, then the result of this operator is new and nothing  else is executed.  And if the object on the left of the operator is not new, the execution continues to the  right.  And when we replace this with a dot, then if a language code is new, we just get a runtime  exception because a new object doesn't have another method called uppercase.  What's the difference?  So we do this mutation and what test do I need to kill this?  Should what?  Should not throw.  Should not throw.  Okay, how do you assert on that?  Okay, well at least I don't know how can I assert that there was no exception but I can  set this variable to new, execute the metro under test and if there is an exception, the  test will fail anyway so I don't have to assert there wasn't an exception but I can  assert that the result was new.  So if the framework asserts whether an exception was not raised, then of course that's a valid  answer but I don't know how to do this with Ruby.  Okay.  It collides if you read the examples as is on the slide.  What I haven't shown is that this variable has a non-new value by default.  So that works then.  But yeah, good catch.  Now let's try and find some bugs with mutation testing.  One bug that I was able to discover just by using mutation testing in a project, not actively  looking for the bug is this.  So we have a class called network which represents the networking settings on your Linux computer  and there's an attribute called self-device which is the name of the network interface  and this method for equality is obviously wrong, as highlighted here and we'll see why.  So if self-device is not on an empty string, that piece of the Boolean expression is always  evaluated to false and the entire return value is always false.  So it doesn't matter what's the other object you're comparing to, this method always returns  false and if self-device isn't non or it's not an empty string, then which means it has  some meaningful value in the software, this is evaluated to true and the Boolean expression  always depends on the second part.  So and that's actually the fix, just remove this first part of the Boolean expression  and here we go.  The reason this statement detected for I guess about seven or eight years is there wasn't  a single test in the test suite which tried to test this equality method when some of  the attributes was an empty string or was none.  So they were always testing with some valid values and this went undetected for many,  many years.  Now also in the software under test, when you see normal conditions, this value is always,  it always has some value so it doesn't return false or something bad happens.  Another book which I was able to find is as shown, I have two classes, the second class  inherits from the first one and you see both classes have parameter code speed limit in  the init method and both parameters have default values.  Now this is perfectly valid Python code so there's nothing wrong with the way it's written.  I run this through mutation testing and I get a surviving mutant.  The reason here is this is a constant change, the two for Python adds plus one to any integer  constant and to see what happens.  And I immediately know the reason for this is that I don't have any test which creates  an object from this class and asserts what the default value of this attribute should  be.  So I create my test like this, just create an object from the class under test and assert  what the default value should be and the test immediately fails of course.  And the reason for this is if you look closely, you see that this parameter, I'm not sending  that to the init method of the parent class so it must go here after self.  And when I'm not doing this, the parent class has its default value so it uses 50 instead  of 90 and that's why my test fails.  And this is also very subtle, it stayed undetected for many, many years and also the reason for  not being detected is the attribute to this value is assigned to, is never used in the  software under test.  It was meant to be used by external clients of this software, the software under test  in this case is a library so it's used by other tools and apparently nobody bothered  to check whether or not the default value is it should be.  Another possible book very close to the previous one is again I have two classes, the second  class inherits from the first one.  Again I have a parameter with the default value and this time notice I am actually sending  this to the init method of the parent class and everything must be fine.  I run this through my mutation testing tool, I get the same surviving mutants so again  I write the same type of test, create an object and assert what the default value of the attribute  should be.  And this time I get another type of exception, it's an attribute error telling me object  from the Mock Pro 8 class does not have an attribute called speed limit.  And at this point I start wondering why this is, so I look here, everything looks cool,  I look here, everything looks okay and I traverse back to the parent class and immediately  here I notice first there's no parameter called speed limit and then something starts to  look not right.  And then I look in the body of the parent init method and nobody cares about whether  or not there's a parameter called speed limit, nobody sets an attribute called self speed  limit.  So that's the problem, you know, I don't have this.  One possible fix is to, if you really need this to just take care of it in the class under  test and set the attribute or another possible fix is to delete everything related to this  parameter, I don't bother with it and that was actually the fix in production.  Another thing which mutation testing is really good at is forcing you to look at your source  code and refactor it and places where mutation testing really shines is where you have if  statements and comparisons and lots of Boolean expressions, stuff like that.  The reason for this is we have many mutations in places like this.  So in this example on the screen we have around 100 different mutations.  Every comparison operator can lead to almost 10 different mutations.  So equals here can be replaced with non-equals with less than or greater than equals in Python  is not and also in and not in operators, so that many.  Boolean n can be replaced with Boolean or we can negate the entire Boolean expressions.  Also in the tool for Ruby you can replace the Boolean expression with a true or false  constant and I think also you can change only parts of it so you can replace this with true  or false and then leave the rest as is.  And the Python 2 doesn't do this at the moment but it's fairly easy to add.  So this goes through mutation testing and something, you know, there are surviving mutants and  when you start looking at it you notice the pattern highlighted in red.  So my thought is I can delete this and move the second block of fifth statements to the  right and it becomes a little bit more clear.  And then I notice another pattern that whatever value is, I'm looking for an attribute under  self-handle with the same name and do something with it.  So I can refine this even further and use the get attribute function and this becomes  like this.  In reality this fits only on four lines instead of ten lines and it's much more easier to  test and much more, you know, easier to read actually.  And that's a good thing.  So we've seen, you know, what mutation testing can do.  It forces you to write better asserts and in my opinion when you have a complex software  under test we not only should assert what the return values of our methods are but we  also should assert what intermediate state or side effects are produced by the functions  under test.  And we all agree that it doesn't matter how much we try to write clean software, we always  have these methods which do more than one thing at a time.  So they do something, some calculations, return some value and oh by the way I've just set  this attribute on the site just so you know.  And that's what mutation testing is really helpful with.  It helps you discover these things which you are not testing by mutating them so you can  write your tests better.  We saw, we can find some interesting bugs and we saw that we can find places which we  can refactor.  And the question still stands, is my test suite good enough?  And another question, another side of this question is how do I measure how good my test  suite is?  Which metric do I use to tell whether or not my test suite is good?  And you know, metrics is fairly controversial topic.  I just want to mention some research that's been going on in the last few years.  In 2015 at GTAC there was one lightning talk which says you know coverage is not a good  metric because it doesn't give you a lot of information so go for mutation score, you  know use mutation testing and measure how much of the mutants are killed.  If you kill 100% of the mutants then you're good.  And then last year at GTAC there was another researcher who basically said well you know  the guys from last year they didn't do really good research, they didn't research on many  software so we did better research and we claimed that the coverage metric like lying  coverage and branch coverage is still the best metric in practice.  They say the problem with mutation testing is it is very expensive to compute and in  their research it gave only additional 4% of information which the guys didn't know  already compared to coverage so they say you know use coverage, don't use mutation testing.  And I decided to do a small experiment and see which one of these researchers is right.  So my software under test is called Pelikan AB and this is a very small library, it's  a plugin for Pelikan.  Pelikan is a static HTML generator for Python which you can use to run your block or your  source, siton.  And Pelikan AB gives you one additional TAC for the templating markup which you can use  to encode a varying layout for your website.  So for example you can change the cover of links or covers of buttons stuff like that.  The way to use this software is to define the AB experiment variable in the shell and  run the make command.  If you run make GitHub then the site gets rendered, everything is published directly  to GitHub.  The way to render several versions of your website is first start with the control version  and then name each experiment by name and run make in a sequence like shown on the screen.  And if you do this, the three commands then you get the control version of your website  on experiment which is called 123 and everything about this experiment goes into a directory  with the same name, the URL structure is updated and you can point your users to only to that  experiment and see how they respond and stuff like that.  And in the version under test we have 100% branch and line coverage of this software  and we also have a book.  There is a setting called delete output directory which is set to true by default and this setting  isn't something the software under test controls.  This lives into an external file where your website configuration goes.  So stuff like your website name, your GitHub handle goes into that file as well.  The bug is when that setting is set to true Pelikan will go and delete the output directory  and delete all the HTML files and then start rendering them in a clean directory.  So the result of this command sequence with the setting set to true is that you have deleted  your entire website and only left with the last one.  And imagine now you delete everything and type make GitHub and everything goes live.  So that's a pretty good way to destroy your website.  So I didn't have a test which will fail if this setting is set to true.  So I decided to integrate mutation testing into the project and I wrote a few more tests  to achieve 100% mutation coverage, mind you this is a very small library, very small plugin  and the bug was still present.  So this means I don't have any test which fails when this setting is set to true.  And I started thinking, okay why I have these many tests and this bug is still present and  the answer is you cannot discover this type of bug without looking at the external environment  and that's why we need integration tests.  So I added an integration test which simulates the external environment with the settings  and simulates the make command then tries to verify what content has been rendered and  whether or not it's correct and the test immediately failed of course.  So then I said I fixed the bug but in reality just changed this setting to false and also  added to check whether or not this is set to true just raise an exception in my software.  And now we have 100% mutation coverage, 100% branch coverage and also we have at least  one integration test and I'm thinking okay I must be good then you know if I have so  many tests and I'm using these techniques then possibly my software is bug free and  of course this is not true.  I've added a pilot to the project and pilot was immediately able to discover this bug.  This is a problem with how we code the super method instead of passing the class name.  I was using a shortcut which is self.underscore class and this works perfectly fine when you  use the software in its intended environment because we have only one class and self class  is evaluated to that particular class's name and everything's fine.  This becomes a problem the moment you try to inherit from the class under test and create  a new class and do something different with it.  Then when you code the class under test init method Python goes into a loop between the  parent and the inherited class init methods.  So if you want to learn more just head on to this pilot pull request number and there's  a very detailed description why this is a problem and I am guilty of using this shortcut  in Python but I've seen this in many many projects online at GitHub and I've seen this  in popular software which is used by many people.  So this tells me people don't have very good understanding of what self class actually means  and how it works and that's why we keep using it in the wrong way.  So to conclude the mutation versus coverage topic this is a link to my blog describing  my experiment with more details and there are links to actual git commits so you can  see what actually what code was changed and how it was changed.  So I think when we first start doing testing what we look at is how much coverage we can  do so we strive to write more tests to test as much as possible of the software under  test and that's a good thing until some time when we go to 100% coverage we don't get any  more information out of this metric so it doesn't do any good.  Then we start looking at mutation testing and mutation testing tells us okay now you  have some coverage and you are testing some stuff but then you know there's a lot more  you can test and this is you know what you need to do and we start doing mutation testing  and we get to 100% mutation coverage and when we get to 100% we don't get any more useful  information out of that and we also need integration tests because of the external environment  as you see from the examples we have different types of environments.  One type of environment is the regular environment which our users will be using and another  possible type of environment is a developer just taking our software and trying to do  something else with it or you know build on top of it and we have no idea what these  environments will be and how people will want to use our software so that's why we need  different types of tests and you know possibly we also use different types of tools to detect  the problems and deal with it and I do need more examples on the topic I will be continuing  to explore this topic throughout the year but if you have examples which you can share  with me or you know publish something to GitHub please send me an email.  Now I will go to something more practical, speed of execution, so as you can imagine  mutation testing is very slow and just to measure how slow I have taken a real world  project from Fedora it's called Pi Kickstart, it is a text parser library used by the Fedora  installation program, it's a medium-sized project with a little over 100 files I think  all of these files are Python modules, they don't have many dependencies between each  of them so that's good.  Each module you know does some checking, a few if statements, there are hardly any loops  in the code so you know very easy to understand actually, there is some text then they write  some text as well as an output and this is a library which is meant to be used by other  programs so it doesn't really do anything useful on its own.  The project has a fairly good test suite over 90% coverage with a lot of tests and the other  good thing about it is the files under the source directory, MapoMOS 1 to 1 with the  files under the test directory so they have the same names.  And first thing I did was take Cosmic Ray, the mutation testing tool for Python and told  it, okay so here's the source directory which means load everything into memory, you can  find under this directory in terms of modules, produce all the possible mutations you can  produce on these modules and then here is the test directory, use the test runner to  discover all possible cases you can discover and just start running and I let this run  on my computer that took over six days.  Then I became smarter and wrote a small script to go through the source directory, take only  one file, load this into Cosmic Ray which means produce mutations but only for that  module, don't bother about the rest of the software, only that module and oh by the way  here is only one file in the test directory which contains your test so use only that  for testing and not everything else.  And that was faster, I've also added an option called fail fast which means whenever one  of the tests fails we know that there was a failure and we'll kill the mutants so don't  bother to execute the rest of the cases, this is an option for the test runner and I did  some refactorings like stuff like if length of string is greater than zero, I've replaced  this with only if string and this helped me save about 1000 mutations which were very  obvious things and I let this run, the execution time was now a little over six hours and this  is 20 times more improvement in speed of execution but still quite slow for any practical purposes  I must say.  So the way to use mutation testing at the moment in my opinion is if you're testing  a very small library or a very small project then you can go full on into CI and just tell  okay this is my common line to schedule my mutation testing jobs and let this run for  10, 15, 20 minutes and that should be fine.  If your project is anything bigger than 200 lines this is not going to work very well but  what you can do is create a commit hook or pull request hook which examines the payload  and you can first thing you can do is schedule mutation testing only against the files which  have been modified but by that particular commit or pull request and that should be  faster.  Next thing you can do depending on the tooling is go down from the module level to the class  name and to the method name which has been modified.  The Python 2 doesn't know anything about classes and modules it only knows, it doesn't  know anything about classes and methods sorry, it only knows about modules so regardless  if you have one class or 1000 classes in one module Python loads the entire module and  starts doing mutations against everything.  On the other hand the Ruby 2 knows about classes and methods I think and you can tell you know  test only that particular stuff and you can try and go even further because this is a  pull request or commit hook you have access to the actual diff and you can apply this  to the source under test and assuming you've tested everything before and it's fine then  you can schedule mutations only against the lines which were changed.  So you can do this and it's not really impractical to do this.  Another thing you can do is go into parallel again depending on the tools the Python 2  is built around salary so you can very easily hook this to some messaging back end like  RabbitMQ and schedule hundreds and thousands of messages and let your infrastructure deploy  Docker instances or virtual machines in the cloud and just run your tests in parallel  and just get back the results and you can do this if you have a lot of money of course.  Here is a list of some mutation testing tools.  I have been using only the first two I mostly use Python and I contribute to Cosmic Ray  for Python and I've used Mutant for Ruby but not very actively at the moment.  The names to the rights are GitHub repositories.  So as far as I know the tools on the top are based on the abstract syntax tree of the language  and they are language specific so whatever you want to do with the tool it works on ASTs  and modifies the nodes of the AST.  One thing I like to do is look at the other tools and see what they're doing especially  in terms of mutation operators and try to bring this to Python.  So if you want to actively use mutation testing I really advise you to look at tools for other  languages and see how they work and how they are doing because many of the tools are very  new they are not very mature and on the bottom this is another tool for mutation testing  called the Mule project which Alex Denisoff will be presenting later today so I'll definitely  be checking this out.  This is an LLVM base tool so it should work for several different languages.  If you are into mutation testing also check this out.  And now before I can go further I can take some questions from the audience if you like.  Okay go.  So the question is if we have a zombie but we have 100% code coverage does it mean that  coverage was wrong?  Yeah.  Yeah okay so it doesn't mean coverage was wrong like I mean it doesn't mean the metric  the measurement was wrong it was probably correct.  There are other problems with coverage like for example if you have one line with the  lots of with a long Boolean expression then it still gets covered regardless of how much  of that expression is evaluated so you may be evaluating only the first part of the expression  and still cover that line but you know the next 10 parts of the Boolean expression are  not evaluated.  Yeah well the way to count you know these cases you should take this up to the authors  of tools who do coverage but there are many many publications online with respect to problems  in coverage and why it's not really a good metric why you shouldn't use coverage and  really for me coverage is a vanity metric it really tells you how much of the you know  of the lines you've covered but nothing more it doesn't tell you anything more and if you  have a zombie on a line which was covered this simply means you might have executed  this line but probably you didn't assert on some condition or you know you assert it on  one condition and you need it to assert on two conditions.  I have a comment to that basically this is a difference between line coverage, statement  coverage and branch coverage.  Yes yes.  100% branch coverage then it's unlikely that you will have 100% coverage and have zombies  but most of the tools count only line coverage and very few counties on statement coverage  of the mentioned.  We have a question there.  Yes so this works really well with primitive operations that might change you know less  than data that plus into plus two.  What if you're not using our pre-tips in business that you know if you're not using library  especially in JavaScript but I don't know if you're using those or your condition is  scoring with functions.  So the question is what if we don't use primitives, we use some library for the business logic  how does mutation testing come into play.  This is very dependent on the tools that you use.  So for example the Python tool until recently it was very poor on mutation operators because  it's a new tool and not many people use it.  On the other hand the Ruby tool called Mutant I think this is one of the best tools currently  in existence and it has tons of mutation operators, tons of conditions.  It understands because it's used by people who work on commercial software they get  paid to actually write test suites for commercial software and they support the tool for that  reason.  About JavaScript I don't really have no idea, I don't use JavaScript but really the take  here is you need to know your tools very well and you need to know the software very well  and that's when you can decide whether or not that's going to be useful.  Maybe in your case you might write a plug-in or an extension to the tool and produce the  mutation based on the functions in that particular library.  Question here?  So thank you for your talk.  Academic claims that the biggest problem of mutation testing is equivalent mutants.  Can you say something about them, how often do you see them in production or is it something  that you struggle with and practice?  So the question was about equivalent mutants.  Equivalent mutants are things which change the code but don't really change behavior  in any practical way.  So for example we may have a less-done operator which in practice is equivalent to less-done  or equal depending on the values we accept in the application.  I don't have any concrete measure about this in practice but I think about 10 or 15 percent  of the time I see equivalent mutants.  Equivalent mutants because the behavior doesn't change, the syntax changes but the behavior  doesn't change, the test suite doesn't fail and we cannot kill them and they just stay  as is.  In the project I use mutation testing for, I usually have some threshold like about 10  percent or something like that.  And if the mutants score is above a certain line then I say we're good and then the CI  system goes green and if you drop under that line then we go red and inspect what's happening  and that threshold is based on how many equivalent mutants I have.  So I try to get some idea about that and then go from there.  I've talked to other people who do mutation testing and have been doing mutation testing  for a long, long time.  And this is a problem but still the benefits you get from asserting on all those different  conditions and looking at your code base and understanding the software much more better  which is the result of mutation testing, it just, you know, the benefit is greater than  having to deal with equivalent mutants.  Thank you.  Okay.  More questions or no questions?  We have two minutes so if there are no more questions I'll just show you this very quickly.  I have started to document my findings about mutation testing so first of all I don't forget  them and some of them make really good examples.  This is available on Read the Docs and it's also available on GitHub if you'd like to  contribute.  And it, since the last week we have also Chinese translation for this and I'll skip the trivial  examples and show you the last one.  So we have this in Python it's fairly common to have methods for equality which compare  two objects by comparing that all of the attributes are equal.  So in this example we have a class called sandwich and you can modify the meat of your  sandwich and the bread type of your sandwich and we say that two sandwiches are equal if  meat and bread attributes are equal and that's about it.  You know, we have a safety check here if the other object is none, the editor falls and  the not equals method is just a negation of the equals.  And the way to test this with mutation testing is like so.  So we create two test objects, sandwich one and sandwich two.  By default all of their attributes have a value of empty string which is not shown on  the screen.  And first thing we do is test the test for equality they should be equal because everything  is an empty string inside these objects and we also assert that the two objects are not,  you know, when compared for non-equality it returns false and that takes care of half  of the testing.  Then we test the safety check compared to none and expect to not be equal and then comes  the fun part.  The way to, so this starts changing into different comparison operators and the way to test is  like this.  So we need to modify only one of the attributes at one of the test objects and leave the rest  of the attributes and the other object unchanged and we need to do this for every single attribute.  So when all the attributes are of the same type you just do this and if they're not of  the same type you can do grouping or something like that to, you know, to modify this block.  That's when the time is up so thank you very much.
Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_0_47.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_48_131.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_132_383.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_384_527.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_528_1199.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_1200_1871.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_1872_2783.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_2784_3503.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_3504_3683.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_3684_4127.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_4128_4211.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_4212_4559.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_4560_4787.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_4788_5447.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_5448_6227.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_6228_7295.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_7296_8447.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_8448_8543.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_8544_11747.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_11748_12107.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_12108_12527.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_12528_13079.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_13080_13271.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_13272_13403.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_13404_13583.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_13584_14087.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_14088_14351.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_14352_14831.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_14832_15371.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_15372_15503.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_15504_16343.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_16344_18011.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_18012_18911.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_18912_18971.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_18972_19607.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_19608_20447.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_20448_21239.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_21240_22319.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_22320_23519.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_23520_23807.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_23808_24407.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_24408_25139.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_25140_26495.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_26496_27335.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_27336_27575.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_27576_27851.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_27852_29963.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_29964_30827.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_30828_31139.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_31140_32471.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_32472_39551.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_39552_42731.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_42732_50279.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_50280_54035.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_54036_56291.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_56292_59423.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_59424_67079.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_67080_67127.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_67128_67175.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_67176_67967.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_67968_68135.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_68136_69275.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_69276_69407.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_69408_70331.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_70332_70403.png Mutants_tests_and_zombies_aka_mutation_testing_with_Python_and_a_pinch_of_Ruby_70404_70448.png
Hello everyone. So my name is Alex. I have been testing open source software for the last 10 years. This is how to reach me if you like. And this is how I look. No pretty close. So today I'm going to talk about mutants and zombies but not the ones from the movie. It's slightly different. And I have some code examples. So first let me get an idea how many of you are familiar with Python at least to some degree. Okay, good. And how many of you are familiar with Ruby? Okay, great. So don't worry. The examples are not very hard to understand. So let's take this piece of code. It's a simple class describing one model in the Rails application. There's only one field that goes into the database. And this is only one method which is actually a really stupid method just returns an uppercase string of that field. And as you can see from this screenshot, this class has 100% code coverage. And the question that I'm asking myself as a tester and the question that I expect everybody should be asking themselves here is, is my test sweet good enough? And good enough in terms whenever there's some change in the software under test, is my test suitable to detect if this change will break something or the change goes undetected and possibly goes into production and something bad really happens? And mutation testing can help you answer this question. But first I need to explain what a mutation is. So a mutation is usually a very small change in the software which somehow changes the behavior of the software. Mutations can come from comparison operator and if statements. And if we take the example on the screen, the mutation is replacing the great done operator with a less than operator. And if you apply this mutation to your code and read it, it becomes if h is less than 18, then by beer. Now imagine that you are an online store and suddenly you start doing this, sell beer to small children and this is not good. Then another possible source of mutations is constant values. You can replace these values with something else. And in this example, we replace truth false and because it's very early in the morning yet, if we didn't have our coffees, we would be looking like this. Another possible source of mutations is loops. We may, for example, modify the loop condition or in this example, we may change break to continue and we get an endless loop. And when you have a endless loop, this is what happens. So this is again an example from production and all my examples are taken from production environments today. Well, not the beer one, but by definition, mutation testing is a very small thing. Mutation testing is a technique and also the tools for mutation testing. They modify your software under test. They don't modify your test suite. You keep the test suite separate and for each mutant that is produced in your software under test, you execute the test suite again and again and again. And this gives you a pretty good idea at the end how good your test suite is. It can tell you places where you have tests, but they don't do a very good job of doing finding all possible things that may change. Or they can also tell you places where you have missing tests, but you already notice because we use coverage. So mutation testing is really good for telling you where you need to make better tests. And the idea behind this is some of the mutations which you see and which the tools support try to mimic errors which developers may do while writing code, for example, plus one, minus one errors, delete something by accident and just commit this to source control, stuff like that. And other mutations which you may see, they are purely artificial, but somehow they help us validate the test conditions, the test environment which we run our test suite in and help us expose something that's missing. The algorithm for mutation testing is actually very simple. So we run three loops, one after the other. First one is we go through the all mutation operators that our tool in particular can support, these are stuff which the tool knows how to change with something else, regardless whatever that else may be. Then for each operator we find the places in the source code where that operator is used and replace it with something else. Most operators and conditions can lead to only one another type of mutation, but sometimes you can produce different mutations for only one place in the code, like with the comparison operators. And then of course you execute the test suite. There are three simple rules to kill mutants which you must remember. First when you execute the test suite against the non-modified version of the program, everything should pass and this is a hard requirement, you cannot go without it. If something fails obviously your software doesn't work or your test suite is broken and you need to take measures. And if you have flake tests and sometimes they pass, sometimes they fail and you have no idea why this is, obviously your mutation results will also be unreliable and you need to take measures to fix your flake tests. The second rule is when you run the test suite against a mutant that is a modified version of your program, you expect the result to be failed and that's a good thing. When the test suite fails then we say that the mutant was killed or the mutant died and this means we had at least one assertion or one condition in the test suite which wasn't met and the test suite failed. So that means at least we have one test which is able to detect this change and tell us the software was modified. And the last thing is which you don't want to happen when you run your test suite against the mutant and the result is pass. This is a bad thing. Then we say we have a zombie or that the mutant survived and as you know from the movie zombies are dating that go around and try to eat you and now imagine you make some change in the code, run the test suite it passes but it doesn't really understand there's anything change and now this change goes into production and suddenly this becomes a bug and tries to eat you. So these are the three rules. Testing against the non-modified version should pass always. Testing against a mutant should fail. This is the way mutation test tools figure out when you kill the mutant. When something fails in the test suite the tool says you kill the mutant. Okay now. Let's play a little game. I am going to show you possible mutations and because you all know about testing you are going to tell me what test cases I need to write to kill the mutants. I am using the code from earlier the one with the 100% coverage and only take this method. See this is a string variable and we return the uppercase of the string variable. That's what it does. So very simple. First possible mutation is instead of returning the uppercase string my method under test returns new value. So what test can you propose so I can kill this? Yep. Yes, obviously not new. Yes, correct. So we execute the method under test and expect the result to obviously not be new. If we apply the mutation and run the test the method will return new and the expectation will fail. Everything will fail and we kill the mutant. Next possible mutation is instead of returning the uppercase string I am returning self because this method is part of the class and I have access to the self object so I can do this. What test do I need here to kill this mutation? OK, type string, correct. So I must be checking what the type of the return value is and if I expect only string then it must be only string and nothing else. Another possible mutation is instead of returning the uppercase I just return the value of this variable as is. So how do I kill this? OK. So I start with a value which is in a lower case and then I execute the method under test and expect this to be in a uppercase. And I use this with just a constant because it's easier for me. No. So this one for example will kill the first and the second zombies. And here if I start with new, OK, let me think. With the new test. Yeah it might kill also the first one. So sometimes yeah one of the test is enough to kill more mutants. But actually this is the way I discovered them and I developed the test for them and actually didn't go back to think whether or not I have some test which is not needed. And the last example from the game is replacing this ampersand. This is the new safety navigation operator in Ruby and I just replace it with a dot. So for the folks who don't work with Ruby, the new safety operator works this way. If the object on the left is new, then the result of this operator is new and nothing else is executed. And if the object on the left of the operator is not new, the execution continues to the right. And when we replace this with a dot, then if a language code is new, we just get a runtime exception because a new object doesn't have another method called uppercase. What's the difference? So we do this mutation and what test do I need to kill this? Should what? Should not throw. Should not throw. Okay, how do you assert on that? Okay, well at least I don't know how can I assert that there was no exception but I can set this variable to new, execute the metro under test and if there is an exception, the test will fail anyway so I don't have to assert there wasn't an exception but I can assert that the result was new. So if the framework asserts whether an exception was not raised, then of course that's a valid answer but I don't know how to do this with Ruby. Okay. It collides if you read the examples as is on the slide. What I haven't shown is that this variable has a non-new value by default. So that works then. But yeah, good catch. Now let's try and find some bugs with mutation testing. One bug that I was able to discover just by using mutation testing in a project, not actively looking for the bug is this. So we have a class called network which represents the networking settings on your Linux computer and there's an attribute called self-device which is the name of the network interface and this method for equality is obviously wrong, as highlighted here and we'll see why. So if self-device is not on an empty string, that piece of the Boolean expression is always evaluated to false and the entire return value is always false. So it doesn't matter what's the other object you're comparing to, this method always returns false and if self-device isn't non or it's not an empty string, then which means it has some meaningful value in the software, this is evaluated to true and the Boolean expression always depends on the second part. So and that's actually the fix, just remove this first part of the Boolean expression and here we go. The reason this statement detected for I guess about seven or eight years is there wasn't a single test in the test suite which tried to test this equality method when some of the attributes was an empty string or was none. So they were always testing with some valid values and this went undetected for many, many years. Now also in the software under test, when you see normal conditions, this value is always, it always has some value so it doesn't return false or something bad happens. Another book which I was able to find is as shown, I have two classes, the second class inherits from the first one and you see both classes have parameter code speed limit in the init method and both parameters have default values. Now this is perfectly valid Python code so there's nothing wrong with the way it's written. I run this through mutation testing and I get a surviving mutant. The reason here is this is a constant change, the two for Python adds plus one to any integer constant and to see what happens. And I immediately know the reason for this is that I don't have any test which creates an object from this class and asserts what the default value of this attribute should be. So I create my test like this, just create an object from the class under test and assert what the default value should be and the test immediately fails of course. And the reason for this is if you look closely, you see that this parameter, I'm not sending that to the init method of the parent class so it must go here after self. And when I'm not doing this, the parent class has its default value so it uses 50 instead of 90 and that's why my test fails. And this is also very subtle, it stayed undetected for many, many years and also the reason for not being detected is the attribute to this value is assigned to, is never used in the software under test. It was meant to be used by external clients of this software, the software under test in this case is a library so it's used by other tools and apparently nobody bothered to check whether or not the default value is it should be. Another possible book very close to the previous one is again I have two classes, the second class inherits from the first one. Again I have a parameter with the default value and this time notice I am actually sending this to the init method of the parent class and everything must be fine. I run this through my mutation testing tool, I get the same surviving mutants so again I write the same type of test, create an object and assert what the default value of the attribute should be. And this time I get another type of exception, it's an attribute error telling me object from the Mock Pro 8 class does not have an attribute called speed limit. And at this point I start wondering why this is, so I look here, everything looks cool, I look here, everything looks okay and I traverse back to the parent class and immediately here I notice first there's no parameter called speed limit and then something starts to look not right. And then I look in the body of the parent init method and nobody cares about whether or not there's a parameter called speed limit, nobody sets an attribute called self speed limit. So that's the problem, you know, I don't have this. One possible fix is to, if you really need this to just take care of it in the class under test and set the attribute or another possible fix is to delete everything related to this parameter, I don't bother with it and that was actually the fix in production. Another thing which mutation testing is really good at is forcing you to look at your source code and refactor it and places where mutation testing really shines is where you have if statements and comparisons and lots of Boolean expressions, stuff like that. The reason for this is we have many mutations in places like this. So in this example on the screen we have around 100 different mutations. Every comparison operator can lead to almost 10 different mutations. So equals here can be replaced with non-equals with less than or greater than equals in Python is not and also in and not in operators, so that many. Boolean n can be replaced with Boolean or we can negate the entire Boolean expressions. Also in the tool for Ruby you can replace the Boolean expression with a true or false constant and I think also you can change only parts of it so you can replace this with true or false and then leave the rest as is. And the Python 2 doesn't do this at the moment but it's fairly easy to add. So this goes through mutation testing and something, you know, there are surviving mutants and when you start looking at it you notice the pattern highlighted in red. So my thought is I can delete this and move the second block of fifth statements to the right and it becomes a little bit more clear. And then I notice another pattern that whatever value is, I'm looking for an attribute under self-handle with the same name and do something with it. So I can refine this even further and use the get attribute function and this becomes like this. In reality this fits only on four lines instead of ten lines and it's much more easier to test and much more, you know, easier to read actually. And that's a good thing. So we've seen, you know, what mutation testing can do. It forces you to write better asserts and in my opinion when you have a complex software under test we not only should assert what the return values of our methods are but we also should assert what intermediate state or side effects are produced by the functions under test. And we all agree that it doesn't matter how much we try to write clean software, we always have these methods which do more than one thing at a time. So they do something, some calculations, return some value and oh by the way I've just set this attribute on the site just so you know. And that's what mutation testing is really helpful with. It helps you discover these things which you are not testing by mutating them so you can write your tests better. We saw, we can find some interesting bugs and we saw that we can find places which we can refactor. And the question still stands, is my test suite good enough? And another question, another side of this question is how do I measure how good my test suite is? Which metric do I use to tell whether or not my test suite is good? And you know, metrics is fairly controversial topic. I just want to mention some research that's been going on in the last few years. In 2015 at GTAC there was one lightning talk which says you know coverage is not a good metric because it doesn't give you a lot of information so go for mutation score, you know use mutation testing and measure how much of the mutants are killed. If you kill 100% of the mutants then you're good. And then last year at GTAC there was another researcher who basically said well you know the guys from last year they didn't do really good research, they didn't research on many software so we did better research and we claimed that the coverage metric like lying coverage and branch coverage is still the best metric in practice. They say the problem with mutation testing is it is very expensive to compute and in their research it gave only additional 4% of information which the guys didn't know already compared to coverage so they say you know use coverage, don't use mutation testing. And I decided to do a small experiment and see which one of these researchers is right. So my software under test is called Pelikan AB and this is a very small library, it's a plugin for Pelikan. Pelikan is a static HTML generator for Python which you can use to run your block or your source, siton. And Pelikan AB gives you one additional TAC for the templating markup which you can use to encode a varying layout for your website. So for example you can change the cover of links or covers of buttons stuff like that. The way to use this software is to define the AB experiment variable in the shell and run the make command. If you run make GitHub then the site gets rendered, everything is published directly to GitHub. The way to render several versions of your website is first start with the control version and then name each experiment by name and run make in a sequence like shown on the screen. And if you do this, the three commands then you get the control version of your website on experiment which is called 123 and everything about this experiment goes into a directory with the same name, the URL structure is updated and you can point your users to only to that experiment and see how they respond and stuff like that. And in the version under test we have 100% branch and line coverage of this software and we also have a book. There is a setting called delete output directory which is set to true by default and this setting isn't something the software under test controls. This lives into an external file where your website configuration goes. So stuff like your website name, your GitHub handle goes into that file as well. The bug is when that setting is set to true Pelikan will go and delete the output directory and delete all the HTML files and then start rendering them in a clean directory. So the result of this command sequence with the setting set to true is that you have deleted your entire website and only left with the last one. And imagine now you delete everything and type make GitHub and everything goes live. So that's a pretty good way to destroy your website. So I didn't have a test which will fail if this setting is set to true. So I decided to integrate mutation testing into the project and I wrote a few more tests to achieve 100% mutation coverage, mind you this is a very small library, very small plugin and the bug was still present. So this means I don't have any test which fails when this setting is set to true. And I started thinking, okay why I have these many tests and this bug is still present and the answer is you cannot discover this type of bug without looking at the external environment and that's why we need integration tests. So I added an integration test which simulates the external environment with the settings and simulates the make command then tries to verify what content has been rendered and whether or not it's correct and the test immediately failed of course. So then I said I fixed the bug but in reality just changed this setting to false and also added to check whether or not this is set to true just raise an exception in my software. And now we have 100% mutation coverage, 100% branch coverage and also we have at least one integration test and I'm thinking okay I must be good then you know if I have so many tests and I'm using these techniques then possibly my software is bug free and of course this is not true. I've added a pilot to the project and pilot was immediately able to discover this bug. This is a problem with how we code the super method instead of passing the class name. I was using a shortcut which is self.underscore class and this works perfectly fine when you use the software in its intended environment because we have only one class and self class is evaluated to that particular class's name and everything's fine. This becomes a problem the moment you try to inherit from the class under test and create a new class and do something different with it. Then when you code the class under test init method Python goes into a loop between the parent and the inherited class init methods. So if you want to learn more just head on to this pilot pull request number and there's a very detailed description why this is a problem and I am guilty of using this shortcut in Python but I've seen this in many many projects online at GitHub and I've seen this in popular software which is used by many people. So this tells me people don't have very good understanding of what self class actually means and how it works and that's why we keep using it in the wrong way. So to conclude the mutation versus coverage topic this is a link to my blog describing my experiment with more details and there are links to actual git commits so you can see what actually what code was changed and how it was changed. So I think when we first start doing testing what we look at is how much coverage we can do so we strive to write more tests to test as much as possible of the software under test and that's a good thing until some time when we go to 100% coverage we don't get any more information out of this metric so it doesn't do any good. Then we start looking at mutation testing and mutation testing tells us okay now you have some coverage and you are testing some stuff but then you know there's a lot more you can test and this is you know what you need to do and we start doing mutation testing and we get to 100% mutation coverage and when we get to 100% we don't get any more useful information out of that and we also need integration tests because of the external environment as you see from the examples we have different types of environments. One type of environment is the regular environment which our users will be using and another possible type of environment is a developer just taking our software and trying to do something else with it or you know build on top of it and we have no idea what these environments will be and how people will want to use our software so that's why we need different types of tests and you know possibly we also use different types of tools to detect the problems and deal with it and I do need more examples on the topic I will be continuing to explore this topic throughout the year but if you have examples which you can share with me or you know publish something to GitHub please send me an email. Now I will go to something more practical, speed of execution, so as you can imagine mutation testing is very slow and just to measure how slow I have taken a real world project from Fedora it's called Pi Kickstart, it is a text parser library used by the Fedora installation program, it's a medium-sized project with a little over 100 files I think all of these files are Python modules, they don't have many dependencies between each of them so that's good. Each module you know does some checking, a few if statements, there are hardly any loops in the code so you know very easy to understand actually, there is some text then they write some text as well as an output and this is a library which is meant to be used by other programs so it doesn't really do anything useful on its own. The project has a fairly good test suite over 90% coverage with a lot of tests and the other good thing about it is the files under the source directory, MapoMOS 1 to 1 with the files under the test directory so they have the same names. And first thing I did was take Cosmic Ray, the mutation testing tool for Python and told it, okay so here's the source directory which means load everything into memory, you can find under this directory in terms of modules, produce all the possible mutations you can produce on these modules and then here is the test directory, use the test runner to discover all possible cases you can discover and just start running and I let this run on my computer that took over six days. Then I became smarter and wrote a small script to go through the source directory, take only one file, load this into Cosmic Ray which means produce mutations but only for that module, don't bother about the rest of the software, only that module and oh by the way here is only one file in the test directory which contains your test so use only that for testing and not everything else. And that was faster, I've also added an option called fail fast which means whenever one of the tests fails we know that there was a failure and we'll kill the mutants so don't bother to execute the rest of the cases, this is an option for the test runner and I did some refactorings like stuff like if length of string is greater than zero, I've replaced this with only if string and this helped me save about 1000 mutations which were very obvious things and I let this run, the execution time was now a little over six hours and this is 20 times more improvement in speed of execution but still quite slow for any practical purposes I must say. So the way to use mutation testing at the moment in my opinion is if you're testing a very small library or a very small project then you can go full on into CI and just tell okay this is my common line to schedule my mutation testing jobs and let this run for 10, 15, 20 minutes and that should be fine. If your project is anything bigger than 200 lines this is not going to work very well but what you can do is create a commit hook or pull request hook which examines the payload and you can first thing you can do is schedule mutation testing only against the files which have been modified but by that particular commit or pull request and that should be faster. Next thing you can do depending on the tooling is go down from the module level to the class name and to the method name which has been modified. The Python 2 doesn't know anything about classes and modules it only knows, it doesn't know anything about classes and methods sorry, it only knows about modules so regardless if you have one class or 1000 classes in one module Python loads the entire module and starts doing mutations against everything. On the other hand the Ruby 2 knows about classes and methods I think and you can tell you know test only that particular stuff and you can try and go even further because this is a pull request or commit hook you have access to the actual diff and you can apply this to the source under test and assuming you've tested everything before and it's fine then you can schedule mutations only against the lines which were changed. So you can do this and it's not really impractical to do this. Another thing you can do is go into parallel again depending on the tools the Python 2 is built around salary so you can very easily hook this to some messaging back end like RabbitMQ and schedule hundreds and thousands of messages and let your infrastructure deploy Docker instances or virtual machines in the cloud and just run your tests in parallel and just get back the results and you can do this if you have a lot of money of course. Here is a list of some mutation testing tools. I have been using only the first two I mostly use Python and I contribute to Cosmic Ray for Python and I've used Mutant for Ruby but not very actively at the moment. The names to the rights are GitHub repositories. So as far as I know the tools on the top are based on the abstract syntax tree of the language and they are language specific so whatever you want to do with the tool it works on ASTs and modifies the nodes of the AST. One thing I like to do is look at the other tools and see what they're doing especially in terms of mutation operators and try to bring this to Python. So if you want to actively use mutation testing I really advise you to look at tools for other languages and see how they work and how they are doing because many of the tools are very new they are not very mature and on the bottom this is another tool for mutation testing called the Mule project which Alex Denisoff will be presenting later today so I'll definitely be checking this out. This is an LLVM base tool so it should work for several different languages. If you are into mutation testing also check this out. And now before I can go further I can take some questions from the audience if you like. Okay go. So the question is if we have a zombie but we have 100% code coverage does it mean that coverage was wrong? Yeah. Yeah okay so it doesn't mean coverage was wrong like I mean it doesn't mean the metric the measurement was wrong it was probably correct. There are other problems with coverage like for example if you have one line with the lots of with a long Boolean expression then it still gets covered regardless of how much of that expression is evaluated so you may be evaluating only the first part of the expression and still cover that line but you know the next 10 parts of the Boolean expression are not evaluated. Yeah well the way to count you know these cases you should take this up to the authors of tools who do coverage but there are many many publications online with respect to problems in coverage and why it's not really a good metric why you shouldn't use coverage and really for me coverage is a vanity metric it really tells you how much of the you know of the lines you've covered but nothing more it doesn't tell you anything more and if you have a zombie on a line which was covered this simply means you might have executed this line but probably you didn't assert on some condition or you know you assert it on one condition and you need it to assert on two conditions. I have a comment to that basically this is a difference between line coverage, statement coverage and branch coverage. Yes yes. 100% branch coverage then it's unlikely that you will have 100% coverage and have zombies but most of the tools count only line coverage and very few counties on statement coverage of the mentioned. We have a question there. Yes so this works really well with primitive operations that might change you know less than data that plus into plus two. What if you're not using our pre-tips in business that you know if you're not using library especially in JavaScript but I don't know if you're using those or your condition is scoring with functions. So the question is what if we don't use primitives, we use some library for the business logic how does mutation testing come into play. This is very dependent on the tools that you use. So for example the Python tool until recently it was very poor on mutation operators because it's a new tool and not many people use it. On the other hand the Ruby tool called Mutant I think this is one of the best tools currently in existence and it has tons of mutation operators, tons of conditions. It understands because it's used by people who work on commercial software they get paid to actually write test suites for commercial software and they support the tool for that reason. About JavaScript I don't really have no idea, I don't use JavaScript but really the take here is you need to know your tools very well and you need to know the software very well and that's when you can decide whether or not that's going to be useful. Maybe in your case you might write a plug-in or an extension to the tool and produce the mutation based on the functions in that particular library. Question here? So thank you for your talk. Academic claims that the biggest problem of mutation testing is equivalent mutants. Can you say something about them, how often do you see them in production or is it something that you struggle with and practice? So the question was about equivalent mutants. Equivalent mutants are things which change the code but don't really change behavior in any practical way. So for example we may have a less-done operator which in practice is equivalent to less-done or equal depending on the values we accept in the application. I don't have any concrete measure about this in practice but I think about 10 or 15 percent of the time I see equivalent mutants. Equivalent mutants because the behavior doesn't change, the syntax changes but the behavior doesn't change, the test suite doesn't fail and we cannot kill them and they just stay as is. In the project I use mutation testing for, I usually have some threshold like about 10 percent or something like that. And if the mutants score is above a certain line then I say we're good and then the CI system goes green and if you drop under that line then we go red and inspect what's happening and that threshold is based on how many equivalent mutants I have. So I try to get some idea about that and then go from there. I've talked to other people who do mutation testing and have been doing mutation testing for a long, long time. And this is a problem but still the benefits you get from asserting on all those different conditions and looking at your code base and understanding the software much more better which is the result of mutation testing, it just, you know, the benefit is greater than having to deal with equivalent mutants. Thank you. Okay. More questions or no questions? We have two minutes so if there are no more questions I'll just show you this very quickly. I have started to document my findings about mutation testing so first of all I don't forget them and some of them make really good examples. This is available on Read the Docs and it's also available on GitHub if you'd like to contribute. And it, since the last week we have also Chinese translation for this and I'll skip the trivial examples and show you the last one. So we have this in Python it's fairly common to have methods for equality which compare two objects by comparing that all of the attributes are equal. So in this example we have a class called sandwich and you can modify the meat of your sandwich and the bread type of your sandwich and we say that two sandwiches are equal if meat and bread attributes are equal and that's about it. You know, we have a safety check here if the other object is none, the editor falls and the not equals method is just a negation of the equals. And the way to test this with mutation testing is like so. So we create two test objects, sandwich one and sandwich two. By default all of their attributes have a value of empty string which is not shown on the screen. And first thing we do is test the test for equality they should be equal because everything is an empty string inside these objects and we also assert that the two objects are not, you know, when compared for non-equality it returns false and that takes care of half of the testing. Then we test the safety check compared to none and expect to not be equal and then comes the fun part. The way to, so this starts changing into different comparison operators and the way to test is like this. So we need to modify only one of the attributes at one of the test objects and leave the rest of the attributes and the other object unchanged and we need to do this for every single attribute. So when all the attributes are of the same type you just do this and if they're not of the same type you can do grouping or something like that to, you know, to modify this block. That's when the time is up so thank you very much.
change in the software under test, is my test answer this question. But first I need to explain statements. And if we take the example on the and this is not good. Then another possible testing is a technique and also the tools for the test suite separate and for each mutant that our tool in particular can support, these are which wasn't met and the test suite failed. So that means at least we have one test which is you are going to tell me what test cases I need Boolean expressions. Also in the tool for Ruby mention some research that's been going on in the we have at least one integration test and I'm and how it works and that's why we keep using it need more examples on the topic I will be was take Cosmic Ray, the mutation testing tool of some mutation testing tools. I have been using contribute to Cosmic Ray for Python and I've used mutation testing and have been doing mutation