Sustainable way of testing your code
Amirov, Eugene
Eugene Amirov - Sustainable way of testing your code How to write a test so you would remember what it does in a year from now? How to write selective tests with different inputs? What is test? How to subclass tests cases and yet maintain control on which tests would run? How to extend or to filter inputs used in parent classes? Are you a tiny bit intrigued now? :) This is not another talk about how to test, but how to organize your tests so they were maintainable. I will be using nose framework as an example, however main ideas should be applicable to any other framework you choose. Explaining how some parts of code works I would have to briefly touch some advanced python topics, although I will provide need-to-know basics there, so people with any level of python knowledge could enjoy the ride.
 Hi. Do I need to address it? Hello. Do you hear me? Again, great. My name is Eugene and I work as a team leader at Scrapping Hub. I was lucky enough over a few past years to work in different areas of IT, like games for social networks or collective blog. Again, like Reddit, but just for Russia. So, major part of my job was to review code and I might say that readability of tests, as we see it now, is not really good enough. But actually tests are very important to our work. If we look at top 100 Mod Start projects at GitHub, we can see that 23% of code is located in test folders. So, this means that if your job is to just read through the code of this project that you would spend two hours a day just reading through tests. And tests become more and more important. If from this 100 projects, 34 projects have revision history as much as five years deep. So we can take a quick look into it. It's a percentage of tests to the code it tests and we see it steadily grown. And if we take a look at some absolute metric, like a number of lines of code, we can see that growth is even more significant. It's from 143,000 of light to nearly half million lines for just tests. So, if we take a look at some general tests, for example, I'm sorry. Yeah. This is how most of tests are written, at least the ones that I see. But when we look at it, we can do better job of describing our intent. That means that when other people would read this code, they wouldn't have such question like what this code does. Is this test actually test what it's supposed to? And if there is an error in the test run, was it because of there is error in the code or because test is written wrong? So what actually is test? How can we understand? All tests consist of three main things. First one is environment. Environment which we have before we test something. It could be no special environment. It could be fixed date and time. It could be if you test in some chat service, we have a profile of user. Another thing is what we actually test. We can test that to multiply by two equals four. We can calculate yesterday eight. We can make these fictional VIP users try to swear in public channel. And after each event, we have some expectations. This means that we expect that two multiplied by two equals four. That yesterday calculated correctly at that. Swerving user would be banned no matter of his status. So nearly a decade ago, father of behavior-driven development then North published an article where he proposed given when then template to describe such behavior. Let's take a look at Wikipedia example. Here first line describe our environment. We have customer who has bought a sweater from us. We have stock with this sweater. We test that when sweater is returned, then number of sweaters at our stock will increase by one. And this is very clearly where we see what we have, what we do, and what we want in result. In behavior-driven development, they are right these tests as a text. So managers without any programming experience could write those. Or even users. But they have additional layer of transforming text to actual code. Benefit from this template, we actually don't need that layer. What we can do is simply extract methods, generate an environment to the methods which start with the word given. And so the same with when and then. And let's have an example. I showed you this test before. What we test here is when we registered a logger for Sentry, it doesn't register same or twice. So same error wouldn't appear twice in the Sentry itself. Let's take a look what is environment here. Creating loggers in environment. So we move it to a separate method. Same for action. We can see that there is two actions. It's a little bit alarming we talk about it later. And what results. So now when we transform code in this way, it's much better to understand what it does. And what we expected of it. Actually, when we have two actions, it's a good alarming thing because we almost never want two actions in our code. Because we want to check that progressing from one state environment with an action, we get to another state. And when second action we do, we want to generate that previous state. And the first action we just moved to a separate test. So I didn't highlight it. I forgot to highlight it. But you can see the second test method self given handler registered. This is where we describe what we had at the beginning of this. Also, when we put it in literal words, we can see when something wrong with what we wanted to test. So here, last expectation of which test, then number of sentry handlers registered. Actually, what we want instead is to check that every sentry handler registered is unique. So we move it to another method. Of course, we change behavior of it. I just put it here. Another thing, when users and developers firstly approach this template, they tend to put everything. And I mean everything in the code itself, so it's read correctly. It's kind of again common sense. We don't need this. We have setup. So move common, don't be afraid. You still check up setup when you read this. So this is a good way to structure single test. But single test is not only our problem in test. We also want to organize better tests with its multiple tests we have better organized. So let's take a look at another example. If you take a look at every aquarium, it has some equipment. It has environment in it like salt water, fresh water. This is a Dutch aquarium where majority of space is taken by plants. So this is kind of like classes. So we can take this example. So here is an example of these aquariums. And we see that they inherited from each other. And also we want to check single method. A lot of methods could be... But this method on the top of class and it's inherited by each, every one of them. So of course with aquarium we have a lot of fish. And usually with such many test we have test like this. It's a lot of repetition and I mean a lot. And there could be hundreds of cases. For example if you test in parcel for its removing malicious code from it. There actually could be hundreds of cases. So we don't want such long tests. Usually it's transformed in a loop. So we have aquarium and for each data we loop through it. We check, we do some action, we test. Same but it's just a little bit better. I can't stress enough how bad loops are for testing. For this particular case, if there is an error with Goopy, a fish. You don't know whether errors just with Goopy, always every other fish, always some particular fish. But this information is very important because you can spend half a day looking into a place where you shouldn't look for this particular problem. Because problem is somewhere else and you would see this if you immediately have results on which test, pass, which don't. Here for example we see that tests pass for Goopy and for Goldfish. And don't pass for Raspberry and for Leopoldy. Your train of sorts could be, how are those connected? Because you sort of something else at the moment. And thinking about it you would be able to find exact spot which unites those particular problem. So how do we want to translate data to separate tests? I very like NOS parameterized model which looks like this. And by the way, if you don't know, this month NOS parameterized updates and actually applies not just for NOS but Pi test and unit test. So you transform your tests. So test methods now have parameters. You can see after self there is a fish parameter. And you write your code, your data in a list decorating this method. Unfortunately, particular for NOS parameterized there is a problem with inheritance. So if we inherit from freshwater aquarium, and noticed the different data. So we changed Drasbora for Grammys and we don't test for Leopoldy. Our expectations could be either we run tests in Dutch aquarium just for this three particular data. Or it unites all of them and Drasbora and Leopoldy added to those three. Unfortunately, what's actually happened is this. We have four tests for each aquarium. And what happened here, so you would understand. We have four tests from freshwater aquarium. We replace three for those we have data. And the last test is just a leftover which is inherited from previous one. So it's fails and it's not something. It should be considered. So what do, how can we apply inheritance to our tests? So our goals would be to have tests for afterized. Like NOS parameterization like NOS is good enough, but it need to be able to deal with inheritance. So inherited test data. Also, we probably don't want to repeat each test each time for the inherited test case. We want it to be defined in parent class and seamlessly used in the child cases. And we don't want to apply, we don't always want to apply all the data from parent cases to the child cases. So we need a way to control this. So these goals could be a region to next environments for parameterized virtualization. We need to apply single test method to different data as much data as we want. For inherited test data and inheritance of test itself, what we need to solve this is just an access to parent class and we can extract data from it. And for control execution what we want is a way to exclude data. So let's take a look what Python tools and approaches could help us to deal with these goals. Decorator is most important part of parameterization. It's used in every approach which I would show. And unfortunately usually it's diminished to just create a new function which puts some code before or later a regional function. But being function over function, the curator is actually transforming a regional function almost to anything. You could see this, for example, skip if decorator of unit test. It transforms a regional function depending on the condition, either to original function or to skip function that raises it. So decorator can be also applied to classes, not only functions. So this is also transformation of original class. So how would we apply decorator to achieve those? Novels parameterized show us very good way to define data. It's clear, it's understandable. So we decorate each test case with data that it requires. But only decoration doesn't create multiple tests. So we need a way to transform this function with assigned data to it to multiple test cases. So here we just assign to the method data in any way we want. And with second decorator we take this class. We have all these methods defining it. And for those methods which define parameterization, we create additional method. With this approach we target test parameterization, but it's not very good for inheritance. It's applicable because you can do this, but for each class which inherited from this particular, from this test case, you need again to reapply this decorator for new test because what we simply did here is instead of some test case, we created some test case with more methods. It's not applying the behavior to its child cases. Another approach would be metaclass, which is a way to configure how you create tests. So typical approach is, and we would use typical approach here, is we have a name, we have basis which is all parent classes of this particular case, and we have a name space which is dictionary of methods, parameters from which we create this class. So we can manipulate with dictionary before we create this class. So we have some tests which have data assigned to it, which are from all of them, and for those who have data, we add some keys and values to this name space. It's seamless and it is working with inheritance. So when you create a child test cases, this behavior is also transformed there. So unfortunately with metaclass, there's also trade-off. It requires that each metaclass is inherited in a direct chain, not a sum-3. So it's not a problem for your single project when you start packaging your things. When there is one metaclass in one package and one metaclass with another package, you're not correlating with each other. Your users could not be able to use both of them. So this approach is useful at some extent, but also... We can also not return some class, but anything we want. Frames is another approach and it's used by NOS parametrized. When you have a tracebook, it's actually a listing of frames and it's kind of name spaces where cursor is... our execution cursor. NOS parametrized take a frame from which its declaration was called and it is inject new method there. And also... so it's... Here name space would be definition of class test freshwater aquarium. And it's transformed like that. It's like we written it this way. I'm a little bit short of time, so I quickly approach last one. My favorite. Before that, here it is very understandable why we can't approach... why we can't get parent from here. Because at that very moment, we don't have class to get parent from. We're just defining the name space. So with frames, we can't target the parent class. So my favorite one is custom loader of unit test. Loaders responsible for gathering tests from your code and creating suits from them. So we don't need actually anything at this point except to mark some tests with data. And we don't have to think about inheritance. What we have here is something like this. We get names, we iterate through all of names, and if it has some data, we extend some... we extend all of this. If it's not, we create usual test case class. Actually, worth mentioning that unit test uses loader to create different instances for each test method it has. So it's not one class with a lot of tests... with a lot of methods with tests, but a lot of instances for this class which tests single method. So it's very straightforward, so you can read through it, and it's very understandable. So here, what we will do, we decorate data, and when we approach actual test... test run, we create additional tests. We don't change anything into class. We just decide if we want this test, if we don't want this test, or if we want to create multiple tests from this one. I didn't mention for previous approaches, control execution, which skipping inherited data. This is because there's two general approaches for them. For those where we have access to parent classes, we can clearly tell what we want to do with the data from parent classes. We can extend it, we can remove some particular data from parent data, we can completely replace it. Another approach which don't need to create different decorators for this, is just to insert in your test body something like skip if, but I like very much JUnit approach where they have assume. So we can have assume here, which would go nicely with given, then, and when. And it would skip test if it's not applicable at the parent, is a child class. So I want quickly to give some mental experiments. I would read out loud a few things and you try to get to yourself feelings about it. So it's not related to test, it's just get and get help. Now SVM, now, first of all, we had before folders with different approach, and changing code live on production. After that, I am very glad that we have get and get help now. And I am very sure that using approaches that I demonstrated today, whatever briefly it was, I'm sorry, would be able for you to create a framework that applied to your particular project that works best for you, that for you and your team, it would be very easy to create tests, it would be no frustration, it would be low maintenance, and you can quickly navigate through it. Thank you. APPLAUSE First part with new functions, it looks like BDD, isn't it? First part with new function, when you change the part of code, with new function, with full name. You mean organizing it in a different given? It is actually very easy, it has some current cases, for example, when you want a width patch, so you can approach it simply like a patch isn't only, can be used with width, but it has start and end, so you can create given function and start patching it, and then you stop all started patches in tests. Yeah, it looks like, do you know about Cacumber, maybe something that this framework, it's Ruby, right? Yes, it's behavior-driven development, but it is another layer which we actually don't need, because some of our projects are small, some are not. Cacumber is great, and we can benefit without using, just clarifying what we want to do from our test. Okay, thanks. Thank you for the question. Last question. Okay, thank you. Hi. So, hi. How you manage, like, if you have a lot of tests, because my previous project ended with having a lot of tests, how you manage to do not create a lot of functions that just either do BDD, but they do something different underneath, and how to maintain a lot of helper code to make the test, because from what you were showing on the slides, you move out something to the different function to have one responsibility, and then test is easy, but, you know, maintaining that. So, you're not only worried for that, but for the length of a single test where you have a lot of... So, from experience, there is a great deal of work that you can do to make sure that you have the best results and that you can do a lot of things, but you can also do a lot of things that are not very good. So, you know, you can do a lot of things that are not very good, but you can do a lot of then-asserts. So, from experience, there is actually no harm to combine those then-function to some single then-function. So, you would have... Then user is banned, and... In this method, you would have... Then it's deactivated. Then it can do this, then it can do this. So, you don't put all of this in the test, but you can do a test, a method that describes it. So, generally, what you want to do is to follow the organization of your code. So, you organize your code somehow, and it's better to follow the same structure in the test. Okay. Okay. Kind of answered my question, but probably I will have more. Okay. Feel free, of course, to talk to me afterwards. Thank you.
 Hi.  Do I need to address it?  Hello.  Do you hear me?  Again, great.  My name is Eugene and I work as a team leader at Scrapping Hub.  I was lucky enough over a few past years to work in different areas of IT, like games  for social networks or collective blog.  Again, like Reddit, but just for Russia.  So, major part of my job was to review code and I might say that readability of tests,  as we see it now, is not really good enough.  But actually tests are very important to our work.  If we look at top 100 Mod Start projects at GitHub, we can see that 23% of code is located  in test folders.  So, this means that if your job is to just read through the code of this project that  you would spend two hours a day just reading through tests.  And tests become more and more important.  If from this 100 projects, 34 projects have revision history as much as five years deep.  So we can take a quick look into it.  It's a percentage of tests to the code it tests and we see it steadily grown.  And if we take a look at some absolute metric, like a number of lines of code, we can see  that growth is even more significant.  It's from 143,000 of light to nearly half million lines for just tests.  So, if we take a look at some general tests, for example, I'm sorry.  Yeah.  This is how most of tests are written, at least the ones that I see.  But when we look at it, we can do better job of describing our intent.  That means that when other people would read this code, they wouldn't have such question  like what this code does.  Is this test actually test what it's supposed to?  And if there is an error in the test run, was it because of there is error in the code  or because test is written wrong?  So what actually is test?  How can we understand?  All tests consist of three main things.  First one is environment.  Environment which we have before we test something.  It could be no special environment.  It could be fixed date and time.  It could be if you test in some chat service, we have a profile of user.  Another thing is what we actually test.  We can test that to multiply by two equals four.  We can calculate yesterday eight.  We can make these fictional VIP users try to swear in public channel.  And after each event, we have some expectations.  This means that we expect that two multiplied by two equals four.  That yesterday calculated correctly at that.  Swerving user would be banned no matter of his status.  So nearly a decade ago, father of behavior-driven development then North  published an article where he proposed given when then template to describe such behavior.  Let's take a look at Wikipedia example.  Here first line describe our environment.  We have customer who has bought a sweater from us.  We have stock with this sweater.  We test that when sweater is returned, then number of sweaters at our stock will increase by one.  And this is very clearly where we see what we have, what we do, and what we want in result.  In behavior-driven development, they are right these tests as a text.  So managers without any programming experience could write those.  Or even users.  But they have additional layer of transforming text to actual code.  Benefit from this template, we actually don't need that layer.  What we can do is simply extract methods, generate an environment to the methods which start with the word given.  And so the same with when and then.  And let's have an example.  I showed you this test before.  What we test here is when we registered a logger for Sentry, it doesn't register same or twice.  So same error wouldn't appear twice in the Sentry itself.  Let's take a look what is environment here.  Creating loggers in environment.  So we move it to a separate method.  Same for action.  We can see that there is two actions.  It's a little bit alarming we talk about it later.  And what results.  So now when we transform code in this way, it's much better to understand what it does.  And what we expected of it.  Actually, when we have two actions, it's a good alarming thing because we almost never want two actions in our code.  Because we want to check that progressing from one state environment with an action, we get to another state.  And when second action we do, we want to generate that previous state.  And the first action we just moved to a separate test.  So I didn't highlight it.  I forgot to highlight it.  But you can see the second test method self given handler registered.  This is where we describe what we had at the beginning of this.  Also, when we put it in literal words, we can see when something wrong with what we wanted to test.  So here, last expectation of which test, then number of sentry handlers registered.  Actually, what we want instead is to check that every sentry handler registered is unique.  So we move it to another method.  Of course, we change behavior of it.  I just put it here.  Another thing, when users and developers firstly approach this template, they tend to put everything.  And I mean everything in the code itself, so it's read correctly.  It's kind of again common sense.  We don't need this.  We have setup.  So move common, don't be afraid.  You still check up setup when you read this.  So this is a good way to structure single test.  But single test is not only our problem in test.  We also want to organize better tests with its multiple tests we have better organized.  So let's take a look at another example.  If you take a look at every aquarium, it has some equipment.  It has environment in it like salt water, fresh water.  This is a Dutch aquarium where majority of space is taken by plants.  So this is kind of like classes.  So we can take this example.  So here is an example of these aquariums.  And we see that they inherited from each other.  And also we want to check single method.  A lot of methods could be...  But this method on the top of class and it's inherited by each, every one of them.  So of course with aquarium we have a lot of fish.  And usually with such many test we have test like this.  It's a lot of repetition and I mean a lot.  And there could be hundreds of cases.  For example if you test in parcel for its removing malicious code from it.  There actually could be hundreds of cases.  So we don't want such long tests.  Usually it's transformed in a loop.  So we have aquarium and for each data we loop through it.  We check, we do some action, we test.  Same but it's just a little bit better.  I can't stress enough how bad loops are for testing.  For this particular case, if there is an error with Goopy, a fish.  You don't know whether errors just with Goopy, always every other fish, always some particular fish.  But this information is very important because you can spend half a day looking into a place where you shouldn't look for this particular problem.  Because problem is somewhere else and you would see this if you immediately have results on which test, pass, which don't.  Here for example we see that tests pass for Goopy and for Goldfish.  And don't pass for Raspberry and for Leopoldy.  Your train of sorts could be, how are those connected?  Because you sort of something else at the moment.  And thinking about it you would be able to find exact spot which unites those particular problem.  So how do we want to translate data to separate tests?  I very like NOS parameterized model which looks like this.  And by the way, if you don't know, this month NOS parameterized updates and actually applies not just for NOS but Pi test and unit test.  So you transform your tests.  So test methods now have parameters.  You can see after self there is a fish parameter.  And you write your code, your data in a list decorating this method.  Unfortunately, particular for NOS parameterized there is a problem with inheritance.  So if we inherit from freshwater aquarium, and noticed the different data.  So we changed Drasbora for Grammys and we don't test for Leopoldy.  Our expectations could be either we run tests in Dutch aquarium just for this three particular data.  Or it unites all of them and Drasbora and Leopoldy added to those three.  Unfortunately, what's actually happened is this.  We have four tests for each aquarium.  And what happened here, so you would understand.  We have four tests from freshwater aquarium.  We replace three for those we have data.  And the last test is just a leftover which is inherited from previous one.  So it's fails and it's not something.  It should be considered.  So what do, how can we apply inheritance to our tests?  So our goals would be to have tests for afterized.  Like NOS parameterization like NOS is good enough, but it need to be able to deal with inheritance.  So inherited test data.  Also, we probably don't want to repeat each test each time for the inherited test case.  We want it to be defined in parent class and seamlessly used in the child cases.  And we don't want to apply, we don't always want to apply all the data from parent cases to the child cases.  So we need a way to control this.  So these goals could be a region to next environments for parameterized virtualization.  We need to apply single test method to different data as much data as we want.  For inherited test data and inheritance of test itself, what we need to solve this is just an access to parent class  and we can extract data from it.  And for control execution what we want is a way to exclude data.  So let's take a look what Python tools and approaches could help us to deal with these goals.  Decorator is most important part of parameterization.  It's used in every approach which I would show.  And unfortunately usually it's diminished to just create a new function which puts some code before or later a regional function.  But being function over function, the curator is actually transforming a regional function almost to anything.  You could see this, for example, skip if decorator of unit test.  It transforms a regional function depending on the condition, either to original function or to skip function that raises it.  So decorator can be also applied to classes, not only functions.  So this is also transformation of original class.  So how would we apply decorator to achieve those?  Novels parameterized show us very good way to define data.  It's clear, it's understandable.  So we decorate each test case with data that it requires.  But only decoration doesn't create multiple tests.  So we need a way to transform this function with assigned data to it to multiple test cases.  So here we just assign to the method data in any way we want.  And with second decorator we take this class.  We have all these methods defining it.  And for those methods which define parameterization, we create additional method.  With this approach we target test parameterization, but it's not very good for inheritance.  It's applicable because you can do this, but for each class which inherited from this particular, from this test case,  you need again to reapply this decorator for new test because what we simply did here is instead of some test case,  we created some test case with more methods.  It's not applying the behavior to its child cases.  Another approach would be metaclass, which is a way to configure how you create tests.  So typical approach is, and we would use typical approach here, is we have a name,  we have basis which is all parent classes of this particular case,  and we have a name space which is dictionary of methods, parameters from which we create this class.  So we can manipulate with dictionary before we create this class.  So we have some tests which have data assigned to it, which are from all of them,  and for those who have data, we add some keys and values to this name space.  It's seamless and it is working with inheritance.  So when you create a child test cases, this behavior is also transformed there.  So unfortunately with metaclass, there's also trade-off.  It requires that each metaclass is inherited in a direct chain, not a sum-3.  So it's not a problem for your single project when you start packaging your things.  When there is one metaclass in one package and one metaclass with another package,  you're not correlating with each other. Your users could not be able to use both of them.  So this approach is useful at some extent, but also...  We can also not return some class, but anything we want.  Frames is another approach and it's used by NOS parametrized.  When you have a tracebook, it's actually a listing of frames and it's kind of name spaces  where cursor is... our execution cursor.  NOS parametrized take a frame from which its declaration was called  and it is inject new method there.  And also... so it's...  Here name space would be definition of class test freshwater aquarium.  And it's transformed like that.  It's like we written it this way.  I'm a little bit short of time, so I quickly approach last one.  My favorite.  Before that, here it is very understandable why we can't approach... why we can't get parent from here.  Because at that very moment, we don't have class to get parent from.  We're just defining the name space.  So with frames, we can't target the parent class.  So my favorite one is custom loader of unit test.  Loaders responsible for gathering tests from your code and creating suits from them.  So we don't need actually anything at this point except to mark some tests with data.  And we don't have to think about inheritance.  What we have here is something like this.  We get names, we iterate through all of names, and if it has some data, we extend some... we extend all of this.  If it's not, we create usual test case class.  Actually, worth mentioning that unit test uses loader to create different instances for each test method it has.  So it's not one class with a lot of tests... with a lot of methods with tests,  but a lot of instances for this class which tests single method.  So it's very straightforward, so you can read through it, and it's very understandable.  So here, what we will do, we decorate data, and when we approach actual test... test run,  we create additional tests.  We don't change anything into class.  We just decide if we want this test, if we don't want this test,  or if we want to create multiple tests from this one.  I didn't mention for previous approaches, control execution, which skipping inherited data.  This is because there's two general approaches for them.  For those where we have access to parent classes,  we can clearly tell what we want to do with the data from parent classes.  We can extend it, we can remove some particular data from parent data,  we can completely replace it.  Another approach which don't need to create different decorators for this,  is just to insert in your test body something like skip if,  but I like very much JUnit approach where they have assume.  So we can have assume here, which would go nicely with given, then, and when.  And it would skip test if it's not applicable at the parent, is a child class.  So I want quickly to give some mental experiments.  I would read out loud a few things and you try to get to yourself feelings about it.  So it's not related to test, it's just get and get help.  Now SVM, now, first of all, we had before folders with different approach,  and changing code live on production.  After that, I am very glad that we have get and get help now.  And I am very sure that using approaches that I demonstrated today,  whatever briefly it was, I'm sorry,  would be able for you to create a framework that applied to your particular project  that works best for you, that for you and your team,  it would be very easy to create tests, it would be no frustration,  it would be low maintenance, and you can quickly navigate through it.  Thank you.  APPLAUSE  First part with new functions, it looks like BDD, isn't it?  First part with new function, when you change the part of code,  with new function, with full name.  You mean organizing it in a different given?  It is actually very easy, it has some current cases,  for example, when you want a width patch,  so you can approach it simply like a patch isn't only,  can be used with width, but it has start and end,  so you can create given function and start patching it,  and then you stop all started patches in tests.  Yeah, it looks like, do you know about Cacumber,  maybe something that this framework, it's Ruby, right?  Yes, it's behavior-driven development,  but it is another layer which we actually don't need,  because some of our projects are small, some are not.  Cacumber is great, and we can benefit without using,  just clarifying what we want to do from our test.  Okay, thanks.  Thank you for the question.  Last question.  Okay, thank you.  Hi.  So, hi.  How you manage, like, if you have a lot of tests,  because my previous project ended with having a lot of tests,  how you manage to do not create a lot of functions  that just either do BDD,  but they do something different underneath,  and how to maintain a lot of helper code to make the test,  because from what you were showing on the slides,  you move out something to the different function  to have one responsibility, and then test is easy,  but, you know, maintaining that.  So, you're not only worried for that,  but for the length of a single test where you have a lot of...  So, from experience, there is a great deal of work  that you can do to make sure that you have the best results  and that you can do a lot of things,  but you can also do a lot of things that are not very good.  So, you know, you can do a lot of things  that are not very good,  but you can do a lot of then-asserts.  So, from experience, there is actually no harm  to combine those then-function to some single then-function.  So, you would have...  Then user is banned, and...  In this method, you would have...  Then it's deactivated.  Then it can do this, then it can do this.  So, you don't put all of this in the test,  but you can do a test, a method that describes it.  So, generally, what you want to do  is to follow the organization of your code.  So, you organize your code somehow,  and it's better to follow the same structure in the test.  Okay.  Okay.  Kind of answered my question, but probably I will have more.  Okay. Feel free, of course, to talk to me afterwards.  Thank you.
Eugene_Amirov_-_Sustainable_way_of_testing_your_code_0_3671.png Eugene_Amirov_-_Sustainable_way_of_testing_your_code_3672_4643.png Eugene_Amirov_-_Sustainable_way_of_testing_your_code_4644_8819.png Eugene_Amirov_-_Sustainable_way_of_testing_your_code_8820_12683.png Eugene_Amirov_-_Sustainable_way_of_testing_your_code_12684_17591.png Eugene_Amirov_-_Sustainable_way_of_testing_your_code_17592_17651.png Eugene_Amirov_-_Sustainable_way_of_testing_your_code_17652_17723.png Eugene_Amirov_-_Sustainable_way_of_testing_your_code_17724_17843.png Eugene_Amirov_-_Sustainable_way_of_testing_your_code_17844_17927.png Eugene_Amirov_-_Sustainable_way_of_testing_your_code_17928_18011.png Eugene_Amirov_-_Sustainable_way_of_testing_your_code_18012_18119.png Eugene_Amirov_-_Sustainable_way_of_testing_your_code_18120_18227.png Eugene_Amirov_-_Sustainable_way_of_testing_your_code_18228_18275.png Eugene_Amirov_-_Sustainable_way_of_testing_your_code_18276_18443.png Eugene_Amirov_-_Sustainable_way_of_testing_your_code_18444_23555.png Eugene_Amirov_-_Sustainable_way_of_testing_your_code_23556_23951.png Eugene_Amirov_-_Sustainable_way_of_testing_your_code_23952_27251.png Eugene_Amirov_-_Sustainable_way_of_testing_your_code_27252_27299.png Eugene_Amirov_-_Sustainable_way_of_testing_your_code_27300_27743.png Eugene_Amirov_-_Sustainable_way_of_testing_your_code_27744_29195.png Eugene_Amirov_-_Sustainable_way_of_testing_your_code_29196_29267.png Eugene_Amirov_-_Sustainable_way_of_testing_your_code_29268_29507.png Eugene_Amirov_-_Sustainable_way_of_testing_your_code_29508_29675.png Eugene_Amirov_-_Sustainable_way_of_testing_your_code_29676_29759.png Eugene_Amirov_-_Sustainable_way_of_testing_your_code_29760_31103.png Eugene_Amirov_-_Sustainable_way_of_testing_your_code_31104_31691.png Eugene_Amirov_-_Sustainable_way_of_testing_your_code_31692_32003.png Eugene_Amirov_-_Sustainable_way_of_testing_your_code_32004_34331.png Eugene_Amirov_-_Sustainable_way_of_testing_your_code_34332_34499.png Eugene_Amirov_-_Sustainable_way_of_testing_your_code_34500_34715.png Eugene_Amirov_-_Sustainable_way_of_testing_your_code_34716_34799.png Eugene_Amirov_-_Sustainable_way_of_testing_your_code_34800_34859.png Eugene_Amirov_-_Sustainable_way_of_testing_your_code_34860_34979.png Eugene_Amirov_-_Sustainable_way_of_testing_your_code_34980_36251.png Eugene_Amirov_-_Sustainable_way_of_testing_your_code_36252_36839.png Eugene_Amirov_-_Sustainable_way_of_testing_your_code_36840_41315.png Eugene_Amirov_-_Sustainable_way_of_testing_your_code_41316_41615.png Eugene_Amirov_-_Sustainable_way_of_testing_your_code_41616_41879.png Eugene_Amirov_-_Sustainable_way_of_testing_your_code_41880_41927.png Eugene_Amirov_-_Sustainable_way_of_testing_your_code_41928_42251.png Eugene_Amirov_-_Sustainable_way_of_testing_your_code_42252_43019.png Eugene_Amirov_-_Sustainable_way_of_testing_your_code_43020_46524.png
Hi. Do I need to address it? Hello. Do you hear me? Again, great. My name is Eugene and I work as a team leader at Scrapping Hub. I was lucky enough over a few past years to work in different areas of IT, like games for social networks or collective blog. Again, like Reddit, but just for Russia. So, major part of my job was to review code and I might say that readability of tests, as we see it now, is not really good enough. But actually tests are very important to our work. If we look at top 100 Mod Start projects at GitHub, we can see that 23% of code is located in test folders. So, this means that if your job is to just read through the code of this project that you would spend two hours a day just reading through tests. And tests become more and more important. If from this 100 projects, 34 projects have revision history as much as five years deep. So we can take a quick look into it. It's a percentage of tests to the code it tests and we see it steadily grown. And if we take a look at some absolute metric, like a number of lines of code, we can see that growth is even more significant. It's from 143,000 of light to nearly half million lines for just tests. So, if we take a look at some general tests, for example, I'm sorry. Yeah. This is how most of tests are written, at least the ones that I see. But when we look at it, we can do better job of describing our intent. That means that when other people would read this code, they wouldn't have such question like what this code does. Is this test actually test what it's supposed to? And if there is an error in the test run, was it because of there is error in the code or because test is written wrong? So what actually is test? How can we understand? All tests consist of three main things. First one is environment. Environment which we have before we test something. It could be no special environment. It could be fixed date and time. It could be if you test in some chat service, we have a profile of user. Another thing is what we actually test. We can test that to multiply by two equals four. We can calculate yesterday eight. We can make these fictional VIP users try to swear in public channel. And after each event, we have some expectations. This means that we expect that two multiplied by two equals four. That yesterday calculated correctly at that. Swerving user would be banned no matter of his status. So nearly a decade ago, father of behavior-driven development then North published an article where he proposed given when then template to describe such behavior. Let's take a look at Wikipedia example. Here first line describe our environment. We have customer who has bought a sweater from us. We have stock with this sweater. We test that when sweater is returned, then number of sweaters at our stock will increase by one. And this is very clearly where we see what we have, what we do, and what we want in result. In behavior-driven development, they are right these tests as a text. So managers without any programming experience could write those. Or even users. But they have additional layer of transforming text to actual code. Benefit from this template, we actually don't need that layer. What we can do is simply extract methods, generate an environment to the methods which start with the word given. And so the same with when and then. And let's have an example. I showed you this test before. What we test here is when we registered a logger for Sentry, it doesn't register same or twice. So same error wouldn't appear twice in the Sentry itself. Let's take a look what is environment here. Creating loggers in environment. So we move it to a separate method. Same for action. We can see that there is two actions. It's a little bit alarming we talk about it later. And what results. So now when we transform code in this way, it's much better to understand what it does. And what we expected of it. Actually, when we have two actions, it's a good alarming thing because we almost never want two actions in our code. Because we want to check that progressing from one state environment with an action, we get to another state. And when second action we do, we want to generate that previous state. And the first action we just moved to a separate test. So I didn't highlight it. I forgot to highlight it. But you can see the second test method self given handler registered. This is where we describe what we had at the beginning of this. Also, when we put it in literal words, we can see when something wrong with what we wanted to test. So here, last expectation of which test, then number of sentry handlers registered. Actually, what we want instead is to check that every sentry handler registered is unique. So we move it to another method. Of course, we change behavior of it. I just put it here. Another thing, when users and developers firstly approach this template, they tend to put everything. And I mean everything in the code itself, so it's read correctly. It's kind of again common sense. We don't need this. We have setup. So move common, don't be afraid. You still check up setup when you read this. So this is a good way to structure single test. But single test is not only our problem in test. We also want to organize better tests with its multiple tests we have better organized. So let's take a look at another example. If you take a look at every aquarium, it has some equipment. It has environment in it like salt water, fresh water. This is a Dutch aquarium where majority of space is taken by plants. So this is kind of like classes. So we can take this example. So here is an example of these aquariums. And we see that they inherited from each other. And also we want to check single method. A lot of methods could be... But this method on the top of class and it's inherited by each, every one of them. So of course with aquarium we have a lot of fish. And usually with such many test we have test like this. It's a lot of repetition and I mean a lot. And there could be hundreds of cases. For example if you test in parcel for its removing malicious code from it. There actually could be hundreds of cases. So we don't want such long tests. Usually it's transformed in a loop. So we have aquarium and for each data we loop through it. We check, we do some action, we test. Same but it's just a little bit better. I can't stress enough how bad loops are for testing. For this particular case, if there is an error with Goopy, a fish. You don't know whether errors just with Goopy, always every other fish, always some particular fish. But this information is very important because you can spend half a day looking into a place where you shouldn't look for this particular problem. Because problem is somewhere else and you would see this if you immediately have results on which test, pass, which don't. Here for example we see that tests pass for Goopy and for Goldfish. And don't pass for Raspberry and for Leopoldy. Your train of sorts could be, how are those connected? Because you sort of something else at the moment. And thinking about it you would be able to find exact spot which unites those particular problem. So how do we want to translate data to separate tests? I very like NOS parameterized model which looks like this. And by the way, if you don't know, this month NOS parameterized updates and actually applies not just for NOS but Pi test and unit test. So you transform your tests. So test methods now have parameters. You can see after self there is a fish parameter. And you write your code, your data in a list decorating this method. Unfortunately, particular for NOS parameterized there is a problem with inheritance. So if we inherit from freshwater aquarium, and noticed the different data. So we changed Drasbora for Grammys and we don't test for Leopoldy. Our expectations could be either we run tests in Dutch aquarium just for this three particular data. Or it unites all of them and Drasbora and Leopoldy added to those three. Unfortunately, what's actually happened is this. We have four tests for each aquarium. And what happened here, so you would understand. We have four tests from freshwater aquarium. We replace three for those we have data. And the last test is just a leftover which is inherited from previous one. So it's fails and it's not something. It should be considered. So what do, how can we apply inheritance to our tests? So our goals would be to have tests for afterized. Like NOS parameterization like NOS is good enough, but it need to be able to deal with inheritance. So inherited test data. Also, we probably don't want to repeat each test each time for the inherited test case. We want it to be defined in parent class and seamlessly used in the child cases. And we don't want to apply, we don't always want to apply all the data from parent cases to the child cases. So we need a way to control this. So these goals could be a region to next environments for parameterized virtualization. We need to apply single test method to different data as much data as we want. For inherited test data and inheritance of test itself, what we need to solve this is just an access to parent class and we can extract data from it. And for control execution what we want is a way to exclude data. So let's take a look what Python tools and approaches could help us to deal with these goals. Decorator is most important part of parameterization. It's used in every approach which I would show. And unfortunately usually it's diminished to just create a new function which puts some code before or later a regional function. But being function over function, the curator is actually transforming a regional function almost to anything. You could see this, for example, skip if decorator of unit test. It transforms a regional function depending on the condition, either to original function or to skip function that raises it. So decorator can be also applied to classes, not only functions. So this is also transformation of original class. So how would we apply decorator to achieve those? Novels parameterized show us very good way to define data. It's clear, it's understandable. So we decorate each test case with data that it requires. But only decoration doesn't create multiple tests. So we need a way to transform this function with assigned data to it to multiple test cases. So here we just assign to the method data in any way we want. And with second decorator we take this class. We have all these methods defining it. And for those methods which define parameterization, we create additional method. With this approach we target test parameterization, but it's not very good for inheritance. It's applicable because you can do this, but for each class which inherited from this particular, from this test case, you need again to reapply this decorator for new test because what we simply did here is instead of some test case, we created some test case with more methods. It's not applying the behavior to its child cases. Another approach would be metaclass, which is a way to configure how you create tests. So typical approach is, and we would use typical approach here, is we have a name, we have basis which is all parent classes of this particular case, and we have a name space which is dictionary of methods, parameters from which we create this class. So we can manipulate with dictionary before we create this class. So we have some tests which have data assigned to it, which are from all of them, and for those who have data, we add some keys and values to this name space. It's seamless and it is working with inheritance. So when you create a child test cases, this behavior is also transformed there. So unfortunately with metaclass, there's also trade-off. It requires that each metaclass is inherited in a direct chain, not a sum-3. So it's not a problem for your single project when you start packaging your things. When there is one metaclass in one package and one metaclass with another package, you're not correlating with each other. Your users could not be able to use both of them. So this approach is useful at some extent, but also... We can also not return some class, but anything we want. Frames is another approach and it's used by NOS parametrized. When you have a tracebook, it's actually a listing of frames and it's kind of name spaces where cursor is... our execution cursor. NOS parametrized take a frame from which its declaration was called and it is inject new method there. And also... so it's... Here name space would be definition of class test freshwater aquarium. And it's transformed like that. It's like we written it this way. I'm a little bit short of time, so I quickly approach last one. My favorite. Before that, here it is very understandable why we can't approach... why we can't get parent from here. Because at that very moment, we don't have class to get parent from. We're just defining the name space. So with frames, we can't target the parent class. So my favorite one is custom loader of unit test. Loaders responsible for gathering tests from your code and creating suits from them. So we don't need actually anything at this point except to mark some tests with data. And we don't have to think about inheritance. What we have here is something like this. We get names, we iterate through all of names, and if it has some data, we extend some... we extend all of this. If it's not, we create usual test case class. Actually, worth mentioning that unit test uses loader to create different instances for each test method it has. So it's not one class with a lot of tests... with a lot of methods with tests, but a lot of instances for this class which tests single method. So it's very straightforward, so you can read through it, and it's very understandable. So here, what we will do, we decorate data, and when we approach actual test... test run, we create additional tests. We don't change anything into class. We just decide if we want this test, if we don't want this test, or if we want to create multiple tests from this one. I didn't mention for previous approaches, control execution, which skipping inherited data. This is because there's two general approaches for them. For those where we have access to parent classes, we can clearly tell what we want to do with the data from parent classes. We can extend it, we can remove some particular data from parent data, we can completely replace it. Another approach which don't need to create different decorators for this, is just to insert in your test body something like skip if, but I like very much JUnit approach where they have assume. So we can have assume here, which would go nicely with given, then, and when. And it would skip test if it's not applicable at the parent, is a child class. So I want quickly to give some mental experiments. I would read out loud a few things and you try to get to yourself feelings about it. So it's not related to test, it's just get and get help. Now SVM, now, first of all, we had before folders with different approach, and changing code live on production. After that, I am very glad that we have get and get help now. And I am very sure that using approaches that I demonstrated today, whatever briefly it was, I'm sorry, would be able for you to create a framework that applied to your particular project that works best for you, that for you and your team, it would be very easy to create tests, it would be no frustration, it would be low maintenance, and you can quickly navigate through it. Thank you. APPLAUSE First part with new functions, it looks like BDD, isn't it? First part with new function, when you change the part of code, with new function, with full name. You mean organizing it in a different given? It is actually very easy, it has some current cases, for example, when you want a width patch, so you can approach it simply like a patch isn't only, can be used with width, but it has start and end, so you can create given function and start patching it, and then you stop all started patches in tests. Yeah, it looks like, do you know about Cacumber, maybe something that this framework, it's Ruby, right? Yes, it's behavior-driven development, but it is another layer which we actually don't need, because some of our projects are small, some are not. Cacumber is great, and we can benefit without using, just clarifying what we want to do from our test. Okay, thanks. Thank you for the question. Last question. Okay, thank you. Hi. So, hi. How you manage, like, if you have a lot of tests, because my previous project ended with having a lot of tests, how you manage to do not create a lot of functions that just either do BDD, but they do something different underneath, and how to maintain a lot of helper code to make the test, because from what you were showing on the slides, you move out something to the different function to have one responsibility, and then test is easy, but, you know, maintaining that. So, you're not only worried for that, but for the length of a single test where you have a lot of... So, from experience, there is a great deal of work that you can do to make sure that you have the best results and that you can do a lot of things, but you can also do a lot of things that are not very good. So, you know, you can do a lot of things that are not very good, but you can do a lot of then-asserts. So, from experience, there is actually no harm to combine those then-function to some single then-function. So, you would have... Then user is banned, and... In this method, you would have... Then it's deactivated. Then it can do this, then it can do this. So, you don't put all of this in the test, but you can do a test, a method that describes it. So, generally, what you want to do is to follow the organization of your code. So, you organize your code somehow, and it's better to follow the same structure in the test. Okay. Okay. Kind of answered my question, but probably I will have more. Okay. Feel free, of course, to talk to me afterwards. Thank you.
there is error in the code or because test is written wrong? So what actually is test? How can development, they are right these tests as a alarming we talk about it later. And what way, it's much better to understand what it does. example. So here is an example of these There actually could be hundreds of cases. So we happened here, so you would understand. We have something. It should be considered. So what do, used in every approach which I would show. And don't have to think about inheritance. What we some data, we extend some... we extend all of to do with the data from parent classes. We can feelings about it. So it's not related to test, behavior-driven development, but it is another something different underneath, and how to because from what you were showing on the slides, organization of your code. So, you organize your