Testable ML Data Science
Peters, Holger
Holger Peters - Using Scikit-Learn's interface for turning Spaghetti Data Science into Maintainable Software Finding a good structure for number-crunching code can be a problem, this especially applies to routines preceding the core algorithms: transformations such as data processing and cleanup, as well as feature construction. With such code, the programmer faces the problem, that their code easily turns into a sequence of highly interdependent operations, which are hard to separate. It can be challenging to test, maintain and reuse such "Data Science Spaghetti code". Scikit-Learn offers a simple yet powerful interface for data science algorithms: the estimator and composite classes (called meta- estimators). By example, I show how clever usage of meta-estimators can encapsulate elaborate machine learning models into a maintainable tree of objects that is both handy to use and simple to test. Looking at examples, I will show how this approach simplifies model development, testing and validation and how this brings together best practices from software engineering as well as data science. _Knowledge of Scikit-Learn is handy but not necessary to follow this talk._
 Thank you for the kind introduction. Hello, I'm Holger Peters from Llyander and I'm presenting to you today how to use scikit learns good interfaces for writing maintainable and testable machine learning code. So this talk will not really focus on the best model development or the best algorithm, it will just show you a way how to structure your code in a way that you can test it and that you can use it in a reliable way in production. For some of you who might not know scikit learn, scikit learn is probably the most well known machine learning package for Python and it's really a great package, it has all batteries included and this is its interface. All right, the problem in general that I'm talking about is that of supervised machine learning in this talk and just imagine a problem. We have on the left side here on the table, we have a table with data, it's a season, that's spring, summer, fall and winter, we have a binary variable encoding whether we have a day that is a holiday or not, each row is a data point and each column is what we call a feature. On the right hand side we have some variable that we'll call a target, it is closely associated to the features and the target is a variable that we would like to predict from our features. Features are known data, targets are data that we want to estimate from a given table on the left. In order to do this, we actually have one data set where we have features and targets matching features and target data and we can use this to train a model and then have a model that predicts. So the interface is as follows, we have a class that represents a machine learning algorithm, it has a method fit that gets features named x and a target array called y and that trains the model, so the model learns about the correlations between features and targets and then we have a method predict that can be called upon the trained estimator and that gives us an estimate y for the given model and the given features x. This is the basic problem of machine learning, there are algorithms to solve this and I'm not going to talk about these algorithms. I rather would like to focus on how to prepare data, the feature data x in this talk and how to make it in a way that is both testable, reliable and readable to software developers and data scientists. If you, I'm sure you want to see how this looks like in a short code snippet and this is actually quite succinct. So in this example here, we generate some data sets, x train, x test and y train, y test, then we create a support vector regressor that is some algorithm that I take off the shelf and so I can learn. We fit the training data and we predict on the test data set and in the end we can obtain a metric, we can test how well is our prediction based on our input features x test. So this is a trained model in scikit-learn, it's very simple, very easy and the big question now is how do we obtain or how can we best prepare input data for our estimator because that table that I showed you might come from an SQL database or from other inputs. It sometimes has to be prepared for the model so we get a good prediction and you can think of this preparation in a way, it's a bit like some preparation as it's done in a factory so there are certain steps that are executed to prepare this data and you have to cut pieces into the right shape so that the algorithm can work with them. One typical preparation that we have for a lot of machine learning algorithms is that of a normally distributed scaling so what we imagine that your data has very high numbers and very low numbers but the algorithm really would like to have values that are nicely distributed around zero with the standard deviation of one. Such a scaling can be easily phrased in Python code so x is an array and we just take the mean over all columns and subtract it from our array x so we subtract the mean of each column from each column and then we calculate based on this we calculate the standard deviation and divide by the standard deviation so now each column should be distributed around a mean of zero now with a standard deviation of approximately one and I prepared a small sample for this so you can see above an input array x and it has two columns. First focus on the right most column that would be a feature variable with values 32, 18, and 31 of course in reality we would have future arrays but for the example a very small one is sufficient and then we apply our scaling and in the end that column now has values that are based around zero and are very close to zero. Now I put in another problem that we have in data processing we have a missing value so just imagine I showed you in the first slide I showed you an example where we have weather data just imagine that the thermometer that measured the temperature was broken on a day so you don't have a value here but you would like that your estimation you would still like an estimation for that day and in such cases we have ways how to fill this data with values and strategies so one strategy is just to replace this not a number value with the mean of this feature variable so you could take the mean of temperatures of historic data to replace such a missing temperature slot and because if you apply our algorithm with the mean and dividing by standard deviation what you'll get is just a yeah in this example you'll get a data error from our code because another number values will just break the mean so I've prepared a bit some code that is does a bit more than our code before so before we just subtracted the mean and divided by the standard deviation and now we would like to replace not a number values by the mean and the reason our code failed before was that taking the mean of a column that contains not a number values numerically just raises not a number so here I replaced our numpy mean function by the function numpy non mean which will yield even with non values in our array x it will yield a proper value for the mean then we can subtract again as we did before the mean and divide by the standard deviation and in the end we'll execute a function numpy non to num which will replace all not a number values by zero and in our rescaled data zero is the mean of the data so we have replaced not a number values by the mean and so how does this new code transform our data and it actually seems to work pretty well the same data example with a new code as a resulting array where both columns are distributed around zero with a small standard deviation and so this is an example of some data processing that you would apply maybe to your data before you feed it into the estimator and yeah this small example actually has a few properties that are very interesting so I said that we if we go back to our example we actually transform our array x and take the standard deviations of all columns and the mean of all columns before we call estimator dot predict but what about the next call when we call estimator dot predict there's also an array x that is feed fed into and we have to process data that goes into this predict accordingly as we have transformed the data that goes into fit why is this because our estimator is has learned about the shapes and correlations of the data that we gave it in fit so the data has to look has to have the same distributions the same shape as the data that it saw during fit and how can we do this how can we make sure that the data has been transformed in the same way and so I could learn has a concept for this and that's the transformer concept a transformer is an object that has this notion of a fit and a transform step so we can fit data fit and as transform we can train it during with a method fit and we can transform it with a method transform and there's a shortcut define as I can learn fit transform that is both at the same time what's important about this is transform returns a modified version of our feature matrix x given a matrix x and during the fit it has to it can also see a y and so now we can actually rephrase our code that did the scaling and not number replacement in terms of such a transformer so I called this I wrote a little class it's called a not number guessing scalar so because it guess us now replacement values for numbers and it scales the data and I implemented a method fit that has this the mean calculation as you can see and it saves the means and the standard deviations of the columns as attributes of the object self and then it has a method transform and transform does the actual transformation it subtracts the mean and it divides by the standard deviation and it replaces not number values by zeros which are zero is the mean of our transformed data and using this pattern we can fit our not number guessing transformer with our training data and then transform the data that we actually would like to use for predict we can transform it in the very same way and another opportunity here is since we have a nicely defined interface and for this we can actually start testing it and I wrote us little tests for our class I think you remember our example array and I create a not number guessing scalar I invoke fit transform to obtain a transformed matrix and then I start testing assumptions that I have about the outcome of this transformation and now the issue that this test actually this this test finds an issue our implementation was wrong because if I calculate the standard deviation for each column and I expect the standard deviation for each column to be one I realize that the standard deviation is not one and that has a very simple reason if we look back at the code I calculate the standard deviation of the input sample before I replace not a number values with zero with the mean so in this example the standard deviation of the input sample is wider than the actual distribution of the data after replacing not a number values with with the mean and because the mean is in the center so and we map not a number values also to the center of the data and that makes the distribution kind of smaller and so in a way if we want to fix this code we have to we have to think about this transform method and the solution is actually to make two transformation steps at first we want to have one transformation step that replaces not number values with the mean and then we want to have a second transformation step that does the actual scaling of the data so we want two transformations and scikit-learn has a nice way to do this it offers ways to compose several transformers several transformations in this case we use a building block and I apologize for the low contrast we use building blocks that are called pipelines a pipeline and a pipeline is a sequential is like a chain of transformers and so during fit when we have when we are training and learning from a feature matrix x we use a first transformator transformator one and invoke fit transform to obtain a transformed version of the data and then we take our second transformer also apply fit transform with the result of the first transformation and finally we will obtain a transformed data set that was transformed by several steps it can have an arbitrary number of transformers in the predict when we have already learned the properties of the data like in our example the mean and the standard deviation we can just invoke transforms and get a transformed x in the end from building a pipeline in scikit-learn we can build them pretty easily there is a make pipeline function and we pass it transformer objects and it will it returns a pipeline object and a pipeline object itself is a transformer that means that it has the fit and the transform method and we can just use it instead of our number guessing scalar that I just presented so we could go back and rewrite this class into two classes one doing the scaling and one doing the number replacement or the question is maybe there is actually some someone has solved this for us already and indeed Python has batteries included and scikit-learn has batteries included so we can actually also use two transformers from scikit-learn's library one of these transformers is called the imputer because imputes missing values and so here number would be replaced by the mean and then we have the standard scalar that scales the data that is distributed in this example represented by the red distribution to one to a data set that is distributed around zero and these two transformers can be joined by a pipeline so here you can see this we just put together the building blocks that we already have we saw make pipeline we use make pipeline here and pass it a imputer instance and a standard scalar instance and then if we fit transform our example array we can actually make sure that our assumption holds true that we would like to have a standard deviation of one we could hear also check for the means and some other tests we have wrapped the data processing with those scikit-learn transformers and we've done this in a way where we can individually test each building block so assume that these were not present in scikit-learn we could just write them ourselves and the tests would be fairly easy and yeah I think that this is the biggest gain that we can have from this so if you're leaving this talk and you want to take something away with it something away from it if you want to write maintainable maintainable software if you want to avoid a spaghetti code and your numeric code try to find ways how to separate different concerns different purposes in your code into independent composable units that you can then combine and you can test them individually you can combine them and then you can make a test for the combined model and that's really a good way to structure your numeric algorithms so in the beginning I showed you an example of a machine learning problem where we just used a machine learning algorithm with an scikit-learn estimator that we fitted and predicted with now I extended this example with a pipeline that does the pre-processing make pipeline we use the imputer we use a standard scaler and we can also add our estimator to this pipeline and now our object S does contain our whole algorithmic pipeline it does contain the pre-processing of the data and it does contain the machine learning code and also it does contain all the fitted and estimated parameters coefficients that are present in our model so we could easily serialize this estimator object using pickle or another serialization library and store it to disk or send it across the world into a different network and then we could load it again restore it and make predictions from it and so to summarize what scikit-learn and these interfaces can do for you and how you should use them we found that it's really beneficial to use this these interfaces that scikit-learn provides for you if you want to write pre-processing code and you can use the fit transform interface for the transformers use them write your own transformers if you don't find those that you need in a library if you write your own transformers try to separate concerns separate responsibilities estimating or scaling your data has nothing to do with correcting other number values so don't put them into the same transformer just write to and compose a new transformer out of the two's for for your model in the end if you keep your transformers and your class is small they are a lot easier to test and if tests fail you will find the issue a lot faster if they are simple and use the features like serialization because you can actually quality control your estimators you can store them you can look at them again in the future it's really handy and in the short time I was not able to tell you everything about the compositional and the testing things that you can do with scikit-learn so I just wanted to give you an outlook on what else you could look at if you want to get into this topic there are tons of other transformers and other meta transformers that compose in scikit-learn that you can take a look at for example a feature union where you can combine different transformers for feature generation and also estimators are composable in scikit-learn so there's a cross validation building block the grid search that actually takes estimators and extends their functionality so their predictions are cross validated according to statistical methods so I'm at the end of my talk I thank you for your attention I'm happy to take questions if you like and if you also if you want to chat with me talk with me you can come up to me anytime hi hi could you please describe your testing environment to use a like a standard library like unit tests and like that too um well basically we we use unit testing frameworks like unit tests or pythes I personally prefer pythes as a test runner and we structure our tests or structure the tests like we would use unit tests in other situations so in the most basic form testing numeric code is not fundamentally different than testing other code it's it's code it has to be tested you have to think of inputs and outputs and you have to structure your code in a way that you don't have to or that in most cases you don't have to do too much work to get a test running and so yeah we have some tools to to generate data and to get more tests that are more going into the direction of integration tests but in general we just use the python tools that non data scientists also use other questions yes that is so if I understood the question correctly the question was if we also apply the transformations to the test data so you're talking about the data that I passed to predict right in the first example not the one that you used for the training the one that's yeah so sorry here you're talking about yeah exactly yes we do this is this is the purpose of splitting in the transformer into those two methods so just pull up the slide again the whole purpose of splitting fit and transform here is that we can repeat repeat this transformation in transform without having to change values for them those estimated parameters mean and standard if we would execute the code and fit again then we would not get the same kind of data into our algorithm that the algorithm expects any other how do you track your model performance over time so in some of our applications we have like data going for for years and we have models that are built up and then for instance that model the assumptions underlying probabilities of the data so we're using mostly be Asian models and the underlying probabilities are changing and we want to revalidate to see how on previous datasets or versions of datasets how the models that are overfitting or underfitting depending on what we have so are you doing anything across versions of datasets to make sure that you know your assumptions aren't missing stuff or adding a new stuff that you have didn't have before okay so you're asking how we actually test the stability of our gene learning models well this is done with cross validation methods and we have for sample datasets we have reference so reference scores and if the reference scores are going getting worse in the future then tests fail basically and then if that happens one has to look into into things why why things are getting worse there's not really a better way than using cross validation methods yeah it's more of a monitoring thing so this talk was more about actually testing testing the code whereas your question was rather about testing the quality of the model so I think these are two different concerns yes they're complementary yeah definitely so I just got curious when you do this what you're working I mean you're working on ipython notebook or do you do it as separate scripts or what do you what do you do for this yeah I'm personally not using ipython notebooks that much I just use I write tests in test files and execute my test runner on them and then use continuous integration and all the tooling that is around unit testing yeah I personally well ipython notebook is no environment snow I much that is really great at exploring things but it's not a environment for test-driven development and so there's no test run an ipython notebook and I personally think all the effort that I put into thinking about some test assertion that I could type into an ipython notebook if I put it into a unit test and check it into my repository it's done continuously over and over again so I really prefer this over extensive use of ipython notebooks I do use it if I want to quickly explore something this is just an add-on so no question your talk was about the testing stuff and this is really great with this modules let's say small units but of course it's also important to have reusability then because then you can really yeah change model or apply to different problems reusing parts of your pipeline any other questions okay thank you thank you very much
 Thank you for the kind introduction.  Hello, I'm Holger Peters from Llyander and I'm presenting to you today how to use scikit  learns good interfaces for writing maintainable and testable machine learning code.  So this talk will not really focus on the best model development or the best algorithm,  it will just show you a way how to structure your code in a way that you can test it and  that you can use it in a reliable way in production.  For some of you who might not know scikit learn, scikit learn is probably the most well  known machine learning package for Python and it's really a great package, it has all  batteries included and this is its interface.  All right, the problem in general that I'm talking about is that of supervised machine  learning in this talk and just imagine a problem.  We have on the left side here on the table, we have a table with data, it's a season,  that's spring, summer, fall and winter, we have a binary variable encoding whether we  have a day that is a holiday or not, each row is a data point and each column is what  we call a feature.  On the right hand side we have some variable that we'll call a target, it is closely associated  to the features and the target is a variable that we would like to predict from our features.  Features are known data, targets are data that we want to estimate from a given table  on the left.  In order to do this, we actually have one data set where we have features and targets  matching features and target data and we can use this to train a model and then have a  model that predicts.  So the interface is as follows, we have a class that represents a machine learning algorithm,  it has a method fit that gets features named x and a target array called y and that trains  the model, so the model learns about the correlations between features and targets and then we have  a method predict that can be called upon the trained estimator and that gives us an estimate  y for the given model and the given features x.  This is the basic problem of machine learning, there are algorithms to solve this and I'm  not going to talk about these algorithms.  I rather would like to focus on how to prepare data, the feature data x in this talk and how  to make it in a way that is both testable, reliable and readable to software developers  and data scientists.  If you, I'm sure you want to see how this looks like in a short code snippet and this  is actually quite succinct.  So in this example here, we generate some data sets, x train, x test and y train, y  test, then we create a support vector regressor that is some algorithm that I take off the  shelf and so I can learn.  We fit the training data and we predict on the test data set and in the end we can obtain  a metric, we can test how well is our prediction based on our input features x test.  So this is a trained model in scikit-learn, it's very simple, very easy and the big question  now is how do we obtain or how can we best prepare input data for our estimator because  that table that I showed you might come from an SQL database or from other inputs.  It sometimes has to be prepared for the model so we get a good prediction and you can think  of this preparation in a way, it's a bit like some preparation as it's done in a factory  so there are certain steps that are executed to prepare this data and you have to cut pieces  into the right shape so that the algorithm can work with them.  One typical preparation that we have for a lot of machine learning algorithms is that  of a normally distributed scaling so what we imagine that your data has very high numbers  and very low numbers but the algorithm really would like to have values that are nicely  distributed around zero with the standard deviation of one.  Such a scaling can be easily phrased in Python code so x is an array and we just take the  mean over all columns and subtract it from our array x so we subtract the mean of each  column from each column and then we calculate based on this we calculate the standard deviation  and divide by the standard deviation so now each column should be distributed around a  mean of zero now with a standard deviation of approximately one and I prepared a small  sample for this so you can see above an input array x and it has two columns.  First focus on the right most column that would be a feature variable with values 32,  18, and 31 of course in reality we would have future arrays but for the example a very small  one is sufficient and then we apply our scaling and in the end that column now has values  that are based around zero and are very close to zero.  Now I put in another problem that we have in data processing we have a missing value  so just imagine I showed you in the first slide I showed you an example where we have  weather data just imagine that the thermometer that measured the temperature was broken on  a day so you don't have a value here but you would like that your estimation you would  still like an estimation for that day and in such cases we have ways how to fill this  data with values and strategies so one strategy is just to replace this not a number value  with the mean of this feature variable so you could take the mean of temperatures of  historic data to replace such a missing temperature slot and because if you apply our algorithm  with the mean and dividing by standard deviation what you'll get is just a yeah in this example  you'll get a data error from our code because another number values will just break the  mean so I've prepared a bit some code that is does a bit more than our code before so  before we just subtracted the mean and divided by the standard deviation and now we would  like to replace not a number values by the mean and the reason our code failed before  was that taking the mean of a column that contains not a number values numerically just  raises not a number so here I replaced our numpy mean function by the function numpy  non mean which will yield even with non values in our array x it will yield a proper value  for the mean then we can subtract again as we did before the mean and divide by the standard  deviation and in the end we'll execute a function numpy non to num which will replace  all not a number values by zero and in our rescaled data zero is the mean of the data  so we have replaced not a number values by the mean and so how does this new code transform  our data and it actually seems to work pretty well the same data example with a new code  as a resulting array where both columns are distributed around zero with a small standard  deviation and so this is an example of some data processing that you would apply maybe  to your data before you feed it into the estimator and yeah this small example actually has a  few properties that are very interesting so I said that we if we go back to our example  we actually transform our array x and take the standard deviations of all columns and  the mean of all columns before we call estimator dot predict but what about the next call when  we call estimator dot predict there's also an array x that is feed fed into and we have  to process data that goes into this predict accordingly as we have transformed the data  that goes into fit why is this because our estimator is has learned about the shapes  and correlations of the data that we gave it in fit so the data has to look has to have  the same distributions the same shape as the data that it saw during fit and how can we  do this how can we make sure that the data has been transformed in the same way and so  I could learn has a concept for this and that's the transformer concept a transformer is an  object that has this notion of a fit and a transform step so we can fit data fit and  as transform we can train it during with a method fit and we can transform it with a  method transform and there's a shortcut define as I can learn fit transform that is both  at the same time what's important about this is transform returns a modified version of  our feature matrix x given a matrix x and during the fit it has to it can also see a  y and so now we can actually rephrase our code that did the scaling and not number replacement  in terms of such a transformer so I called this I wrote a little class it's called a  not number guessing scalar so because it guess us now replacement values for numbers and  it scales the data and I implemented a method fit that has this the mean calculation as  you can see and it saves the means and the standard deviations of the columns as attributes  of the object self and then it has a method transform and transform does the actual transformation  it subtracts the mean and it divides by the standard deviation and it replaces not number  values by zeros which are zero is the mean of our transformed data and using this pattern  we can fit our not number guessing transformer with our training data and then transform  the data that we actually would like to use for predict we can transform it in the very  same way and another opportunity here is since we have a nicely defined interface and for  this we can actually start testing it and I wrote us little tests for our class I think  you remember our example array and I create a not number guessing scalar I invoke fit  transform to obtain a transformed matrix and then I start testing assumptions that I have  about the outcome of this transformation and now the issue that this test actually this  this test finds an issue our implementation was wrong because if I calculate the standard  deviation for each column and I expect the standard deviation for each column to be one  I realize that the standard deviation is not one and that has a very simple reason if we  look back at the code I calculate the standard deviation of the input sample before I replace  not a number values with zero with the mean so in this example the standard deviation  of the input sample is wider than the actual distribution of the data after replacing not  a number values with with the mean and because the mean is in the center so and we map not  a number values also to the center of the data and that makes the distribution kind  of smaller and so in a way if we want to fix this code we have to we have to think about  this transform method and the solution is actually to make two transformation steps  at first we want to have one transformation step that replaces not number values with  the mean and then we want to have a second transformation step that does the actual scaling  of the data so we want two transformations and scikit-learn has a nice way to do this  it offers ways to compose several transformers several transformations in this case we use  a building block and I apologize for the low contrast we use building blocks that are called  pipelines a pipeline and a pipeline is a sequential is like a chain of transformers  and so during fit when we have when we are training and learning from a feature matrix  x we use a first transformator transformator one and invoke fit transform to obtain a transformed  version of the data and then we take our second transformer also apply fit transform with  the result of the first transformation and finally we will obtain a transformed data  set that was transformed by several steps it can have an arbitrary number of transformers  in the predict when we have already learned the properties of the data like in our example  the mean and the standard deviation we can just invoke transforms and get a transformed  x in the end from building a pipeline in scikit-learn we can build them pretty easily there is a  make pipeline function and we pass it transformer objects and it will it returns a pipeline  object and a pipeline object itself is a transformer that means that it has the fit  and the transform method and we can just use it instead of our number guessing scalar that  I just presented so we could go back and rewrite this class into two classes one doing the  scaling and one doing the number replacement or the question is maybe there is actually  some someone has solved this for us already and indeed Python has batteries included and  scikit-learn has batteries included so we can actually also use two transformers from  scikit-learn's library one of these transformers is called the imputer because imputes missing  values and so here number would be replaced by the mean and then we have the standard  scalar that scales the data that is distributed in this example represented by the red distribution  to one to a data set that is distributed around zero and these two transformers can be joined  by a pipeline so here you can see this we just put together the building blocks that  we already have we saw make pipeline we use make pipeline here and pass it a imputer instance  and a standard scalar instance and then if we fit transform our example array we can  actually make sure that our assumption holds true that we would like to have a standard  deviation of one we could hear also check for the means and some other tests we have  wrapped the data processing with those scikit-learn transformers and we've done this in a way where  we can individually test each building block so assume that these were not present in scikit-learn  we could just write them ourselves and the tests would be fairly easy and yeah I think  that this is the biggest gain that we can have from this so if you're leaving this talk  and you want to take something away with it something away from it if you want to write  maintainable maintainable software if you want to avoid a spaghetti code and your numeric  code try to find ways how to separate different concerns different purposes in your code into  independent composable units that you can then combine and you can test them individually  you can combine them and then you can make a test for the combined model and that's  really a good way to structure your numeric algorithms so in the beginning I showed you  an example of a machine learning problem where we just used a machine learning algorithm  with an scikit-learn estimator that we fitted and predicted with now I extended this example  with a pipeline that does the pre-processing make pipeline we use the imputer we use a  standard scaler and we can also add our estimator to this pipeline and now our object S does  contain our whole algorithmic pipeline it does contain the pre-processing of the data  and it does contain the machine learning code and also it does contain all the fitted and  estimated parameters coefficients that are present in our model so we could easily serialize  this estimator object using pickle or another serialization library and store it to disk  or send it across the world into a different network and then we could load it again restore  it and make predictions from it and so to summarize what scikit-learn and these interfaces  can do for you and how you should use them we found that it's really beneficial to use  this these interfaces that scikit-learn provides for you if you want to write pre-processing  code and you can use the fit transform interface for the transformers use them write your own  transformers if you don't find those that you need in a library if you write your own  transformers try to separate concerns separate responsibilities estimating or scaling your  data has nothing to do with correcting other number values so don't put them into the same  transformer just write to and compose a new transformer out of the two's for for your  model in the end if you keep your transformers and your class is small they are a lot easier  to test and if tests fail you will find the issue a lot faster if they are simple and  use the features like serialization because you can actually quality control your estimators  you can store them you can look at them again in the future it's really handy and in the  short time I was not able to tell you everything about the compositional and the testing things  that you can do with scikit-learn so I just wanted to give you an outlook on what else  you could look at if you want to get into this topic there are tons of other transformers  and other meta transformers that compose in scikit-learn that you can take a look at for  example a feature union where you can combine different transformers for feature generation  and also estimators are composable in scikit-learn so there's a cross validation building block  the grid search that actually takes estimators and extends their functionality so their predictions  are cross validated according to statistical methods so I'm at the end of my talk I thank  you for your attention I'm happy to take questions if you like and if you also if you want to  chat with me talk with me you can come up to me anytime hi hi could you please describe  your testing environment to use a like a standard library like unit tests and like that too  um well basically we we use unit testing frameworks like unit tests or pythes I personally prefer  pythes as a test runner and we structure our tests or structure the tests like we would  use unit tests in other situations so in the most basic form testing numeric code is not  fundamentally different than testing other code it's it's code it has to be tested you  have to think of inputs and outputs and you have to structure your code in a way that  you don't have to or that in most cases you don't have to do too much work to get a test  running and so yeah we have some tools to to generate data and to get more tests that  are more going into the direction of integration tests but in general we just use the python  tools that non data scientists also use other questions  yes that is so if I understood the question correctly the question was if we also apply  the transformations to the test data so you're talking about the data that I passed to predict  right in the first example not the one that you used for the training the one that's yeah  so sorry here you're talking about yeah exactly yes we do this is this is the purpose of  splitting in the transformer into those two methods so just pull up the slide again the  whole purpose of splitting fit and transform here is that we can repeat repeat this transformation  in transform without having to change values for them those estimated parameters mean and  standard if we would execute the code and fit again then we would not get the same kind  of data into our algorithm that the algorithm expects any other  how do you track your model performance over time so in some of our applications we have  like data going for for years and we have models that are built up and then for instance  that model the assumptions underlying probabilities of the data so we're using mostly be Asian  models and the underlying probabilities are changing and we want to revalidate to see how  on previous datasets or versions of datasets how the models that are overfitting or underfitting  depending on what we have so are you doing anything across versions of datasets to make  sure that you know your assumptions aren't missing stuff or adding a new stuff that you  have didn't have before okay so you're asking how we actually test the stability of our  gene learning models well this is done with cross validation methods and we have for sample  datasets we have reference so reference scores and if the reference scores are going getting  worse in the future then tests fail basically and then if that happens one has to look into  into things why why things are getting worse there's not really a better way than using  cross validation methods yeah it's more of a monitoring thing so this talk was more about  actually testing testing the code whereas your question was rather about testing the  quality of the model so I think these are two different concerns yes they're complementary  yeah definitely so I just got curious when you do this what you're working I mean you're  working on ipython notebook or do you do it as separate scripts or what do you what do  you do for this yeah I'm personally not using ipython notebooks that much I just use I write  tests in test files and execute my test runner on them and then use continuous integration  and all the tooling that is around unit testing yeah I personally well ipython notebook is  no environment snow I much that is really great at exploring things but it's not a  environment for test-driven development and so there's no test run an ipython notebook  and I personally think all the effort that I put into thinking about some test assertion  that I could type into an ipython notebook if I put it into a unit test and check it  into my repository it's done continuously over and over again so I really prefer this  over extensive use of ipython notebooks I do use it if I want to quickly explore something  this is just an add-on so no question your talk was about the testing stuff and this  is really great with this modules let's say small units but of course it's also important  to have reusability then because then you can really yeah change model or apply to different  problems reusing parts of your pipeline any other questions okay thank you thank you very  much
Using_Scikit-Learn_s_interface_for_turning_Spaghetti_Data_Science_into_Maintainable_Software_0_47.png Using_Scikit-Learn_s_interface_for_turning_Spaghetti_Data_Science_into_Maintainable_Software_48_707.png Using_Scikit-Learn_s_interface_for_turning_Spaghetti_Data_Science_into_Maintainable_Software_708_3503.png Using_Scikit-Learn_s_interface_for_turning_Spaghetti_Data_Science_into_Maintainable_Software_3504_5207.png Using_Scikit-Learn_s_interface_for_turning_Spaghetti_Data_Science_into_Maintainable_Software_5208_5267.png Using_Scikit-Learn_s_interface_for_turning_Spaghetti_Data_Science_into_Maintainable_Software_5268_6599.png Using_Scikit-Learn_s_interface_for_turning_Spaghetti_Data_Science_into_Maintainable_Software_6600_12767.png Using_Scikit-Learn_s_interface_for_turning_Spaghetti_Data_Science_into_Maintainable_Software_12768_12959.png Using_Scikit-Learn_s_interface_for_turning_Spaghetti_Data_Science_into_Maintainable_Software_12960_13079.png Using_Scikit-Learn_s_interface_for_turning_Spaghetti_Data_Science_into_Maintainable_Software_13080_14723.png Using_Scikit-Learn_s_interface_for_turning_Spaghetti_Data_Science_into_Maintainable_Software_14724_17267.png Using_Scikit-Learn_s_interface_for_turning_Spaghetti_Data_Science_into_Maintainable_Software_17268_17699.png Using_Scikit-Learn_s_interface_for_turning_Spaghetti_Data_Science_into_Maintainable_Software_17700_27287.png Using_Scikit-Learn_s_interface_for_turning_Spaghetti_Data_Science_into_Maintainable_Software_27288_29039.png Using_Scikit-Learn_s_interface_for_turning_Spaghetti_Data_Science_into_Maintainable_Software_29040_29339.png Using_Scikit-Learn_s_interface_for_turning_Spaghetti_Data_Science_into_Maintainable_Software_29340_29555.png Using_Scikit-Learn_s_interface_for_turning_Spaghetti_Data_Science_into_Maintainable_Software_29556_29735.png Using_Scikit-Learn_s_interface_for_turning_Spaghetti_Data_Science_into_Maintainable_Software_29736_29939.png Using_Scikit-Learn_s_interface_for_turning_Spaghetti_Data_Science_into_Maintainable_Software_29940_30071.png Using_Scikit-Learn_s_interface_for_turning_Spaghetti_Data_Science_into_Maintainable_Software_30072_30119.png Using_Scikit-Learn_s_interface_for_turning_Spaghetti_Data_Science_into_Maintainable_Software_30120_30599.png Using_Scikit-Learn_s_interface_for_turning_Spaghetti_Data_Science_into_Maintainable_Software_30600_39959.png Using_Scikit-Learn_s_interface_for_turning_Spaghetti_Data_Science_into_Maintainable_Software_39960_41003.png Using_Scikit-Learn_s_interface_for_turning_Spaghetti_Data_Science_into_Maintainable_Software_41004_41615.png Using_Scikit-Learn_s_interface_for_turning_Spaghetti_Data_Science_into_Maintainable_Software_41616_42371.png Using_Scikit-Learn_s_interface_for_turning_Spaghetti_Data_Science_into_Maintainable_Software_42372_46175.png Using_Scikit-Learn_s_interface_for_turning_Spaghetti_Data_Science_into_Maintainable_Software_46176_46235.png Using_Scikit-Learn_s_interface_for_turning_Spaghetti_Data_Science_into_Maintainable_Software_46236_46319.png Using_Scikit-Learn_s_interface_for_turning_Spaghetti_Data_Science_into_Maintainable_Software_46320_48335.png Using_Scikit-Learn_s_interface_for_turning_Spaghetti_Data_Science_into_Maintainable_Software_48336_49274.png
Thank you for the kind introduction. Hello, I'm Holger Peters from Llyander and I'm presenting to you today how to use scikit learns good interfaces for writing maintainable and testable machine learning code. So this talk will not really focus on the best model development or the best algorithm, it will just show you a way how to structure your code in a way that you can test it and that you can use it in a reliable way in production. For some of you who might not know scikit learn, scikit learn is probably the most well known machine learning package for Python and it's really a great package, it has all batteries included and this is its interface. All right, the problem in general that I'm talking about is that of supervised machine learning in this talk and just imagine a problem. We have on the left side here on the table, we have a table with data, it's a season, that's spring, summer, fall and winter, we have a binary variable encoding whether we have a day that is a holiday or not, each row is a data point and each column is what we call a feature. On the right hand side we have some variable that we'll call a target, it is closely associated to the features and the target is a variable that we would like to predict from our features. Features are known data, targets are data that we want to estimate from a given table on the left. In order to do this, we actually have one data set where we have features and targets matching features and target data and we can use this to train a model and then have a model that predicts. So the interface is as follows, we have a class that represents a machine learning algorithm, it has a method fit that gets features named x and a target array called y and that trains the model, so the model learns about the correlations between features and targets and then we have a method predict that can be called upon the trained estimator and that gives us an estimate y for the given model and the given features x. This is the basic problem of machine learning, there are algorithms to solve this and I'm not going to talk about these algorithms. I rather would like to focus on how to prepare data, the feature data x in this talk and how to make it in a way that is both testable, reliable and readable to software developers and data scientists. If you, I'm sure you want to see how this looks like in a short code snippet and this is actually quite succinct. So in this example here, we generate some data sets, x train, x test and y train, y test, then we create a support vector regressor that is some algorithm that I take off the shelf and so I can learn. We fit the training data and we predict on the test data set and in the end we can obtain a metric, we can test how well is our prediction based on our input features x test. So this is a trained model in scikit-learn, it's very simple, very easy and the big question now is how do we obtain or how can we best prepare input data for our estimator because that table that I showed you might come from an SQL database or from other inputs. It sometimes has to be prepared for the model so we get a good prediction and you can think of this preparation in a way, it's a bit like some preparation as it's done in a factory so there are certain steps that are executed to prepare this data and you have to cut pieces into the right shape so that the algorithm can work with them. One typical preparation that we have for a lot of machine learning algorithms is that of a normally distributed scaling so what we imagine that your data has very high numbers and very low numbers but the algorithm really would like to have values that are nicely distributed around zero with the standard deviation of one. Such a scaling can be easily phrased in Python code so x is an array and we just take the mean over all columns and subtract it from our array x so we subtract the mean of each column from each column and then we calculate based on this we calculate the standard deviation and divide by the standard deviation so now each column should be distributed around a mean of zero now with a standard deviation of approximately one and I prepared a small sample for this so you can see above an input array x and it has two columns. First focus on the right most column that would be a feature variable with values 32, 18, and 31 of course in reality we would have future arrays but for the example a very small one is sufficient and then we apply our scaling and in the end that column now has values that are based around zero and are very close to zero. Now I put in another problem that we have in data processing we have a missing value so just imagine I showed you in the first slide I showed you an example where we have weather data just imagine that the thermometer that measured the temperature was broken on a day so you don't have a value here but you would like that your estimation you would still like an estimation for that day and in such cases we have ways how to fill this data with values and strategies so one strategy is just to replace this not a number value with the mean of this feature variable so you could take the mean of temperatures of historic data to replace such a missing temperature slot and because if you apply our algorithm with the mean and dividing by standard deviation what you'll get is just a yeah in this example you'll get a data error from our code because another number values will just break the mean so I've prepared a bit some code that is does a bit more than our code before so before we just subtracted the mean and divided by the standard deviation and now we would like to replace not a number values by the mean and the reason our code failed before was that taking the mean of a column that contains not a number values numerically just raises not a number so here I replaced our numpy mean function by the function numpy non mean which will yield even with non values in our array x it will yield a proper value for the mean then we can subtract again as we did before the mean and divide by the standard deviation and in the end we'll execute a function numpy non to num which will replace all not a number values by zero and in our rescaled data zero is the mean of the data so we have replaced not a number values by the mean and so how does this new code transform our data and it actually seems to work pretty well the same data example with a new code as a resulting array where both columns are distributed around zero with a small standard deviation and so this is an example of some data processing that you would apply maybe to your data before you feed it into the estimator and yeah this small example actually has a few properties that are very interesting so I said that we if we go back to our example we actually transform our array x and take the standard deviations of all columns and the mean of all columns before we call estimator dot predict but what about the next call when we call estimator dot predict there's also an array x that is feed fed into and we have to process data that goes into this predict accordingly as we have transformed the data that goes into fit why is this because our estimator is has learned about the shapes and correlations of the data that we gave it in fit so the data has to look has to have the same distributions the same shape as the data that it saw during fit and how can we do this how can we make sure that the data has been transformed in the same way and so I could learn has a concept for this and that's the transformer concept a transformer is an object that has this notion of a fit and a transform step so we can fit data fit and as transform we can train it during with a method fit and we can transform it with a method transform and there's a shortcut define as I can learn fit transform that is both at the same time what's important about this is transform returns a modified version of our feature matrix x given a matrix x and during the fit it has to it can also see a y and so now we can actually rephrase our code that did the scaling and not number replacement in terms of such a transformer so I called this I wrote a little class it's called a not number guessing scalar so because it guess us now replacement values for numbers and it scales the data and I implemented a method fit that has this the mean calculation as you can see and it saves the means and the standard deviations of the columns as attributes of the object self and then it has a method transform and transform does the actual transformation it subtracts the mean and it divides by the standard deviation and it replaces not number values by zeros which are zero is the mean of our transformed data and using this pattern we can fit our not number guessing transformer with our training data and then transform the data that we actually would like to use for predict we can transform it in the very same way and another opportunity here is since we have a nicely defined interface and for this we can actually start testing it and I wrote us little tests for our class I think you remember our example array and I create a not number guessing scalar I invoke fit transform to obtain a transformed matrix and then I start testing assumptions that I have about the outcome of this transformation and now the issue that this test actually this this test finds an issue our implementation was wrong because if I calculate the standard deviation for each column and I expect the standard deviation for each column to be one I realize that the standard deviation is not one and that has a very simple reason if we look back at the code I calculate the standard deviation of the input sample before I replace not a number values with zero with the mean so in this example the standard deviation of the input sample is wider than the actual distribution of the data after replacing not a number values with with the mean and because the mean is in the center so and we map not a number values also to the center of the data and that makes the distribution kind of smaller and so in a way if we want to fix this code we have to we have to think about this transform method and the solution is actually to make two transformation steps at first we want to have one transformation step that replaces not number values with the mean and then we want to have a second transformation step that does the actual scaling of the data so we want two transformations and scikit-learn has a nice way to do this it offers ways to compose several transformers several transformations in this case we use a building block and I apologize for the low contrast we use building blocks that are called pipelines a pipeline and a pipeline is a sequential is like a chain of transformers and so during fit when we have when we are training and learning from a feature matrix x we use a first transformator transformator one and invoke fit transform to obtain a transformed version of the data and then we take our second transformer also apply fit transform with the result of the first transformation and finally we will obtain a transformed data set that was transformed by several steps it can have an arbitrary number of transformers in the predict when we have already learned the properties of the data like in our example the mean and the standard deviation we can just invoke transforms and get a transformed x in the end from building a pipeline in scikit-learn we can build them pretty easily there is a make pipeline function and we pass it transformer objects and it will it returns a pipeline object and a pipeline object itself is a transformer that means that it has the fit and the transform method and we can just use it instead of our number guessing scalar that I just presented so we could go back and rewrite this class into two classes one doing the scaling and one doing the number replacement or the question is maybe there is actually some someone has solved this for us already and indeed Python has batteries included and scikit-learn has batteries included so we can actually also use two transformers from scikit-learn's library one of these transformers is called the imputer because imputes missing values and so here number would be replaced by the mean and then we have the standard scalar that scales the data that is distributed in this example represented by the red distribution to one to a data set that is distributed around zero and these two transformers can be joined by a pipeline so here you can see this we just put together the building blocks that we already have we saw make pipeline we use make pipeline here and pass it a imputer instance and a standard scalar instance and then if we fit transform our example array we can actually make sure that our assumption holds true that we would like to have a standard deviation of one we could hear also check for the means and some other tests we have wrapped the data processing with those scikit-learn transformers and we've done this in a way where we can individually test each building block so assume that these were not present in scikit-learn we could just write them ourselves and the tests would be fairly easy and yeah I think that this is the biggest gain that we can have from this so if you're leaving this talk and you want to take something away with it something away from it if you want to write maintainable maintainable software if you want to avoid a spaghetti code and your numeric code try to find ways how to separate different concerns different purposes in your code into independent composable units that you can then combine and you can test them individually you can combine them and then you can make a test for the combined model and that's really a good way to structure your numeric algorithms so in the beginning I showed you an example of a machine learning problem where we just used a machine learning algorithm with an scikit-learn estimator that we fitted and predicted with now I extended this example with a pipeline that does the pre-processing make pipeline we use the imputer we use a standard scaler and we can also add our estimator to this pipeline and now our object S does contain our whole algorithmic pipeline it does contain the pre-processing of the data and it does contain the machine learning code and also it does contain all the fitted and estimated parameters coefficients that are present in our model so we could easily serialize this estimator object using pickle or another serialization library and store it to disk or send it across the world into a different network and then we could load it again restore it and make predictions from it and so to summarize what scikit-learn and these interfaces can do for you and how you should use them we found that it's really beneficial to use this these interfaces that scikit-learn provides for you if you want to write pre-processing code and you can use the fit transform interface for the transformers use them write your own transformers if you don't find those that you need in a library if you write your own transformers try to separate concerns separate responsibilities estimating or scaling your data has nothing to do with correcting other number values so don't put them into the same transformer just write to and compose a new transformer out of the two's for for your model in the end if you keep your transformers and your class is small they are a lot easier to test and if tests fail you will find the issue a lot faster if they are simple and use the features like serialization because you can actually quality control your estimators you can store them you can look at them again in the future it's really handy and in the short time I was not able to tell you everything about the compositional and the testing things that you can do with scikit-learn so I just wanted to give you an outlook on what else you could look at if you want to get into this topic there are tons of other transformers and other meta transformers that compose in scikit-learn that you can take a look at for example a feature union where you can combine different transformers for feature generation and also estimators are composable in scikit-learn so there's a cross validation building block the grid search that actually takes estimators and extends their functionality so their predictions are cross validated according to statistical methods so I'm at the end of my talk I thank you for your attention I'm happy to take questions if you like and if you also if you want to chat with me talk with me you can come up to me anytime hi hi could you please describe your testing environment to use a like a standard library like unit tests and like that too um well basically we we use unit testing frameworks like unit tests or pythes I personally prefer pythes as a test runner and we structure our tests or structure the tests like we would use unit tests in other situations so in the most basic form testing numeric code is not fundamentally different than testing other code it's it's code it has to be tested you have to think of inputs and outputs and you have to structure your code in a way that you don't have to or that in most cases you don't have to do too much work to get a test running and so yeah we have some tools to to generate data and to get more tests that are more going into the direction of integration tests but in general we just use the python tools that non data scientists also use other questions yes that is so if I understood the question correctly the question was if we also apply the transformations to the test data so you're talking about the data that I passed to predict right in the first example not the one that you used for the training the one that's yeah so sorry here you're talking about yeah exactly yes we do this is this is the purpose of splitting in the transformer into those two methods so just pull up the slide again the whole purpose of splitting fit and transform here is that we can repeat repeat this transformation in transform without having to change values for them those estimated parameters mean and standard if we would execute the code and fit again then we would not get the same kind of data into our algorithm that the algorithm expects any other how do you track your model performance over time so in some of our applications we have like data going for for years and we have models that are built up and then for instance that model the assumptions underlying probabilities of the data so we're using mostly be Asian models and the underlying probabilities are changing and we want to revalidate to see how on previous datasets or versions of datasets how the models that are overfitting or underfitting depending on what we have so are you doing anything across versions of datasets to make sure that you know your assumptions aren't missing stuff or adding a new stuff that you have didn't have before okay so you're asking how we actually test the stability of our gene learning models well this is done with cross validation methods and we have for sample datasets we have reference so reference scores and if the reference scores are going getting worse in the future then tests fail basically and then if that happens one has to look into into things why why things are getting worse there's not really a better way than using cross validation methods yeah it's more of a monitoring thing so this talk was more about actually testing testing the code whereas your question was rather about testing the quality of the model so I think these are two different concerns yes they're complementary yeah definitely so I just got curious when you do this what you're working I mean you're working on ipython notebook or do you do it as separate scripts or what do you what do you do for this yeah I'm personally not using ipython notebooks that much I just use I write tests in test files and execute my test runner on them and then use continuous integration and all the tooling that is around unit testing yeah I personally well ipython notebook is no environment snow I much that is really great at exploring things but it's not a environment for test-driven development and so there's no test run an ipython notebook and I personally think all the effort that I put into thinking about some test assertion that I could type into an ipython notebook if I put it into a unit test and check it into my repository it's done continuously over and over again so I really prefer this over extensive use of ipython notebooks I do use it if I want to quickly explore something this is just an add-on so no question your talk was about the testing stuff and this is really great with this modules let's say small units but of course it's also important to have reusability then because then you can really yeah change model or apply to different problems reusing parts of your pipeline any other questions okay thank you thank you very much
Holger Peters from Llyander and I'm presenting to machine learning code. So this talk will not really focus on the best model development or the best algorithm, it will just show you a way how to structure your code in a way that you can test that can be called upon the trained estimator and problem of machine learning, there are algorithms talk and how to make it in a way that is both trained model in scikit-learn, it's very simple, but for the example a very small one is example of some data processing that you would the estimator and yeah this small example transformations in this case we use a building transformer objects and it will it returns a you can see this we just put together the maintainable software if you want to avoid a that's really a good way to structure your and you can use the fit transform interface for transformers try to separate concerns separate other transformers and other meta transformers look at for example a feature union where you can takes estimators and extends their functionality describe your testing environment to use a like a of our gene learning models well this is done ipython notebook or do you do it as separate type into an ipython notebook if I put it into a