Spatiotemporal machine learning in Python (Part 1)
Parente, Leandro
Diemen, Chris van
Software requirements: opengeohub/py-geo docker image (gdal, rasterio, eumap, scikit-learn) This tutorial covers the theoretical background for machine learning and python implementations, as well as integrating raster data with scikit-learn models. Why use pyeumap.LandMapper? The tutorial shows how to prepare the training samples with spatial overlay, how to evaluate the ML model performance with spatial cross-validation, how to tune the ML model with hyperparameter optimization, how to get the final ML model ,and finally how to generate spatial predictions using the fitted model.
 Let's back to the training sessions. So this in this afternoon, we will use two slots to present about the spatial temporal machine learn in Python and how we implemented it in the context of the open data science Europe. So the presentation will be, we will start with me and later Chris will join to do some code demonstrations as we are doing for all the sessions. And we'll have a break and after that we will back and we will discuss more some about the ensemble machine learning. So and the, to the approach that we implemented now in the EU map. So before to go to the code, I put some slides. So I prepared these slides to provide some conceptual background about machine learn. So how we it's important to provide like a common understanding of what we are doing in a general way, because of course machine learning can be used by different applications. It's kind of buzzword now. So everybody's talking about it. So considering this general introduction, I will explain how to do it specifically in using geospatial data. So what are the, what is the approach? What are the data? How we can implement a processing workflow to do it? And specifically on that, I will show two demos, two real use applications that classify a land cover. One, it's purely spatial. So that I implemented in my PhD. And now it's spatial temporal and I implemented, we are implementing it in the open data science Europe. So I see it as a kind of evolution between this purely spatial to spatial temporal. So there are some important aspects I will explain about it. And after this introduction, we will have a hands on like a demonstration and we will actually we will execute the code using mainly the EU map package. So there is a class that we've created, landmapper, where you can implement all these machine learning processing specifically for geospatial data. So let's start. So I like to, I would like to start with the definition. So what is machine learning, and if you think in a nutshell, it's essentially a subfield of artificial intelligence. So it's not the same thing. You have more subfields inside of artificial intelligence, like robotics and fuzz logic. So machine learning, it's a subfield inside of it. But most important, it use techniques to learn from the data and make accurate, accurate predictions. So the whole point to use machine learning is train over like some specific data and do predictions in new data. In the field, we call it like a generalization. So you receive new data that the model didn't see and you produce good predictions. And there are several ways to implement it, different algorithms, but in essence, it is. And other important part is without being explicit programed. So if you think what it's happening inside of machine learning model, you can code like hard code, all these rules. So if you think like a developer, I could get an image and do a lot of rules myself to try to identify, for example, where is water and work with the data, try to find some values. And I can code that. But if you think in the way that machine learn do it, it implements it for you. So it will create all these parameters, look in the data and try analyzing the data and try to establish it like automatically. So without being explicit program it. So if you think in the artificial intelligence field, you have other subfields, so machine learning is one of these subfields. And inside of machine learning, you have like other buzzword today, like deep learning. So here we are, we will not talk about deep learning. So we need to, we could have other session about it, but we will use TensorFlow. So it's not every time that you use TensorFlow that you implement deep learning. But and today all these concepts are kind of mixed. So for example, in artificial intelligence, you have like a natural processing language subfield. And today you have pretty good algorithms and really accurate algorithms in deep learn to deal with that. So some of these subfields are really changing too much. And deep learning, it's a really hot field in the research. And there are so many algorithms and really nice implementations of it. But mainly you, when we are talking about deep learning, we are talking like processing graphical card interface, processing unit like GPU, convolution neural networks and all, it's more related with it. We have like a recurrent neural networks also. But here at the end of the training session, we will use a regular artificial neural network and it will run on forest and with the gradate and boosted trees. So in this session, we are actually inside of the, just of the machine learning, not actual deep learning. So now if you think in the machine learning, you have mainly three like categories about algorithms. So we have the supervised, the unsupervised and the reinforcement. So the supervised, it's mainly when you provide like training data and using this training data, you already provide like the answer. So you need to have like raw data can be image can be text can be a table. It doesn't matter. But you need to label this data. So what it means, it means that you will look this data. So some human being will look this data and it will create like what is the expected result for this data. So if you think like in an image, we need to provide for the satellite images, where are the class that we want to map or the values that we want to map. So for example, you could provide like coordinates points, specifically on top of the image showing that this is a patchery, this is a conifer forest and things like that. Or you could provide polygons also. But you need to provide it. So you need to have a data with label, so label data to do the training and the machine learn will try to match and to analyze all this data to provide like expected results and you can compare with that. So here mainly we will use it. You have the unsupervised learner. It's more like machine learning. So you don't need like training data and you can work with the raw data and try to find some like hidden structure on it. So for example, for the spatial classification, you could try analyze purely the values of that image and try to aggregate values that are similar. So if you think maybe a pixel in a forest, it's in specific part of the image, it's close and or it's related has values related with pixels on other part that are forest also. So it's possible. But sometimes the most part of the times actually they are not really accurate. So the accuracy, it's better if you have the label data. But it's there, there are researching, there is a lot of research in this field also and you have the reinforcement learner. So machine learn this reinforce you learn by mistake. So you have like actually an agent that try to do something and you have some way to measure what to analyze if that action produce or not the expected result. And it will learn considering the response of different attempts. So I'm not familiar with this approach and you again, you can combine it so you can have like a supervised learner and you can after that you can have like a reinforcement learn. So but it's important to be aware of these these fields and these concepts. And here, mainly we will work with the supervised machine learn a specific classification algorithm. So we are not doing regression. And if you think in a classification, the difference between classification regression mainly we are producing like factor values or like integer values or class values. And in the regression you are the model will predict like a continuum value. So it's more like to predict some variable. So in our case, the land cover, it's actually like a class, we have the same exactly the same value. It's an integer, but we have the same value for different pixels of the images. But for example, if you think if we we could train a model to predict the precipitation, for example, and to predict precipitation, the output will be like a continuous value. Each pixel could have a some level of the precipitation and the this is more like a continual value. And if you need to use a regression to implement it. So and the other part it's it's about it the program. So inside of the the model, you will have like what we call like the parameters. So the parameters are actually the how the model will implement all these rules, all the training rules to split and to analyze the data and predict the expected result. So to predict if that pixel, it's a class, I'm talking about image, but for example, you can do the same thing for text. So now in nowadays, for example, the Google translate do it. So it analyzed some of the whole structure of the text and predict the output as a new in another language. And it's mainly it's the same concept. So you have the input data set, it's actually text, and you are predicting and the Google translate predicts it as a as a text, but in other language. So and if you think in the way that it was implemented, any anybody created like some anyone created some hide and rules inside of it, the model learned it considering millions of training data in the case of the Google translate, but you can have it for all the for different kinds of applications. And one important concept is when we are talking about the model, the parameters are like. We train the parameters. So if you think in the random forest, it's basically a decision tree. So you will analyze the data and this data could have like eight, 10 features, 10 variables. In our case, it's 10 images. We will use 84, but it could be like any number of covariates and features and analyze this picture, this this features and this pixels. The model will train like a tree and each part of these three, it's a decision tree, it will like try to define to divide this data to classify in pasturing in conifers forest. So you the way that this decision tree is created, this is the like specifically the rules. These are the parameters. The hyper parameters are mainly the they are related with how you control the learning process. So for example, if we use 100 trees or 10 trees for for the random forest, this is the hyper parameter. So number of trees is a hyper parameter inside of each of these trees. We have parameters that will analyze the data and it will split it and it will generate the output. So this is the the parameters of each learner and different algorithms, they have different parameters. But mainly what you need to remember is the parameter, it's actually they are learned from the data. So and it's it's it's automatically derived considering it's a data driven process. So you don't need code anything inside of it. You just need to code outside to control the process. So this is a expected machine learn supervised machine learn approach. So we have some raw data using this raw data, you can have features. Again, this could be anything tables, text, images doesn't matter. And you have labels labels are like, what are they? What is the expected result? Considering this raw data. And of course, the raw data, you can derive just some features, you can have like a bunch of raw data, but just part of it you are using to do machine learning. And you have like the model training and the model evaluation. And with a training model. So after the training, you have like a model, a training model, you can save it, you can load it. But mainly you can use this training model to do prediction over new data. So that all that effort creating the labels and create the expected result. And there it needs to be currently actually it needs to be done by someone. So human human being and me. And you can use it to predict and to generate predictions. So this is the expected workflow, but for the real world applications, we have more something like that. So it's not so beautiful, but you have the when we are talking about real applications, you have like a way more complex, for example, data and raw data than this benchmark data sets. And when I'm talking about benchmark data sets, I'm saying like, if you download, for example, the TensorFlow, you have like some digit numbers, like images with digit numbers, and they are really organized and pretty and ready to go. So you just need to get this data and do some predictions and you already have the labels. But the most part of real applications, they don't have the data really organized like that. You need to prepare the input data set, you need to prepare the labels, you need to expect to prepare the expected result. And you need to work all with all the modeling part. But at the end, you need to do predictions. So otherwise, you are not interesting, just create a model and use this model to just to predicting just a portion of the data. What is the end goal considering a real use application here? And for example, in our case, the end goal, it's really mapping the length over, map the length over classification and the continent of Europe. So we did all this effort, but at the end, we need to predict it. So we need to get this, this model and really scale it up and predict all the pixels of Europe. So there is a way more complexity than just like a simple workflow. But mainly, if I could like put the two most important aspects, you need to prepare the data before to really start doing modeling, starting to machine learn. And you need to be aware what you plan to do with the final model. So what will be your prediction strategy? So you can put it in a, I don't know, in a service and the users can access the servers as for example, we have for the Google Translate, it's a service for a machine learning model. And everybody can go there and just put some text and it will translate. So considering a real use application, you need to be aware of how you will serve your predictions. So you can do all the predictions in-house as we are doing, generate the maps and put it available to the people who use the maps or we can really serve the model and to the other users really access and do new predictions. So there are several possibilities on it and it's important to be aware of these possibilities. So but let's back to our simple example and we can expand it for the geo-spatial data. So now if we specifically explain it in the context of the geo-spatial data, this is more or less the pretty way I say it. So you have like the raw earth observation data. So this earth observation data as I was explaining the morning, so you can have like the Landsat data, Sentinel data, there are a lot of satellite images available, but you need to download it, you need to organize, you need to remove the clouds, you need to do several pre-processing steps and to produce analysis ready earth observation data. So and using this analysis ready earth observation data, you actually need to provide now the labels. So what you want to classify in this image. So you want to classify like soil organic carbon, you want to classify land cover, you want to classify, I don't know, just one class, pastry maybe, or you want to classify water. So you need to provide the labels because the performance of the most, the better models that are available now, they work with supervised classification. So you need to provide the samples, the training samples. And the training samples, it can be like points, lines, polygons, it doesn't matter. A sample, it's just like a vector that you will overlay with the data, get the pixels and put inside of the machine learning. So here in the training session, we will show the like an overlay of data, but in the U-Map, so look at, did talk about it in U-Map, you have like the functions to, I don't know, get one point that samples that you collected in field, for example, and overlay these points over like raster data, different raster layers. So we are, we, in these training sessions, we already did it, so we will work just with the overlay of data, point data, that for each point, we extracted all the pixel values. So and considering it, we will train models, so we need to train a model, we need to evaluate this model, so there are some strategies to do it. And after that, we will have like a trained model, and when I say like trained model, it's a ready to go model, so it's a model, it's a file, it's a model that it will be able to do predictions without do the training again. So we will show it for you in the code. So mainly when you do the training, you have like a model, and this model, he has like, he knows how to process new data and predict like the land cover. So and he did learn considering just the samples, but you can use to do predictions in different parts of your data, of course. So there are some limitations with that, but then go with, is you really, for example, train a model to do the prediction in the whole year. So as we are doing here, and we will, and you can load this model and you can skip all this training process and just predict. Yeah. Yeah. If it's straight going, that's the idea that you will be able to process analysis, right? Rather than the route. No, that's a good question. Yeah. So actually, you need this rule, this arrow here, you need to read from the analysis ready data. So because, yeah, I will fix it in these slides. Thank you for noticing it because this, the training, it's really sensitive. So if you, if you change a bit the pixels, you will change completely the predictions. So it's, it's, there are some techniques to really try to put, like you can simulate noise in the analysis data and to try to predict raw data, but it's, it's more, it's a different type of research. And here we are predicting analysis ready data. So this is one of one example. So I did it in the, my PhD. So for, I worked in Brazil using the Landsat data. So mainly I, we had like more than 30,000 points for each year. And these points were visually interpreted by three specialists and they classified as pastry, forest. So and they are, you can see the points here. They are really spread all over the country. So this is a, we, and in my, in my PhD, I was part of the GIS and remote sense laboratory in Brazil. And actually we generated this data from scratch. So here we actually develop all this. We develop a sampling design. We spread the points all over the country and we develop a solution to help the specialists to check the pixel values and see if it's pastry, if it's crop and things like that. So in this context, it's, it's, it's not like it's really specific, but the, what it's interesting here, we have samples for the whole country and for the, all the years. And that's, it's not the main part of the applications. So, but even have, so considering that we have points for all the years and for all the country, you can see here, this little square, little squares here are like, those are the Landsat scenes. So we have 308 tiles, Landsat scenes to cover the entire country. And we have 34 years. And what we did here were really, we, we, we, we trained one model for each scene. So if you think in the machine learned way, the training process occurred by each Landsat scene and by each year. So at the end, we trained more than 2000, almost 13,000 random forest models. And it, and we mapped individually all the years. So, but if you think we have all this data, so it would be possible training one single model. And what I mean with that, I mean, I can get all the points from all the years and for the whole country and put all these points in a single machine learning model and train it. So it will be millions of points. And but maybe it could produce a better result. And but they're mainly there are two problems with that. We need to have the capacity, software and hard to do it. So it's not easy training a machine learning with millions of points. And we need to have our monizer data. And if you remember, my first in the morning, I explained that if you work with just with the raw data with the lens, we have three sensors. These sensors are different. So one image of 285 will not have the same pixel values, the same reflectance values from like an image of 2020, accurate by Landsat eight. So you need to work with harmonize it, the Earth observation data. So and they are true products at least probably there are more. But the main products to do it, it's the glad Landsat R&D. I talked about it during the morning and there is this other product that combines landsat and sent in out shoe. So in produce like one harmonized data cube, but it's just with Landsat eight. So we will have just since 2013, but I think they process it just from 2015. So this is mainly this is mandatory to try to create one single model for all this to classify for example, multiple years and different parts of Europe. So in the open data science Europe, we are doing it mainly. So we took all the Lucas points and that it's a basically a well-known field survey here in Europe. And we combine it with some generated points from the old from from the other land cover maps Korean land cover classification. And mainly we are using all these points to train one single model, but it's important to emphasize that we have the capacity to do it. And we also have like a harmonized earth observation data as the glad Landsat R&D. And without gaps and all these parts that I was explained during the morning. So at the end, we are predicting dominant class probabilities and uncertainty. So in the end of this training session parts to you will produce it also in the Python and notebook. But we are also using more than one learner. So random forest, it's already a sample learners because inside of a random forest, we have several trees. So decision trees and it's why they call it like a forest. It's multiple trees. But we are also using the gradient boosted trees and the artificial neural network. So we will show how to do it in Python for the tiles that we prepared. And yeah, now it's time to go to the code and start to really do all classifying the land cover, considering the data that we prepared. So just to overview of what we will do mainly, we will train a regular random forest, pretty straightforward without any type of hyperparameter optimization. It will be just a purely random forest and we will predict it to see how it works and to understand how we map it's doing it internally. Later the second example will be the random forest with hyperparameter optimization. So you can test different combinations of hyperparameters to really predict it and benchmark all these different parameters and find what is the best one considering some evaluation metric. And the second part, we will train the random forest with probabilities. So if you think in the random forest, you have multiple trees. So each tree will produce like an output and you can use this output to derive like a kind of a probability. So and this probability, it will, it changes completely the output of the algorithm. So we will see it, but instead to just have a hard class, a dominant class, so it's pastury, forest, you can have like multiple probabilities and these probabilities will show, for example, maybe there are some, there is an area where you have like a not a high probability, you can have a high probability just for one class. So you have sure that the model is pretty sure that this is, for example, coniferal forest, but you can have like other pixel where the coniferal forest is 60% and the broadleaf forest is 40%, 30%. So there is some confusion there. And if you think in the image, man, it kind of makes sense because you had some limitations with the data that we are working. It's not possible really get one pixel and have around 100% sure that inside of that pixel we have just one single length over class because we can have in a transition zone. So there are several complications of it. And at the end, we will expand this and put more learners, so more three learners and explain how we are deriving the uncertainty and how we can improve this result. And this is actually our current workflow. The code that we will show here is actually the same code that we are running in our servers to produce the length over map available in the open data science Europe. So now I will pass the floor to Chris. I don't know if there is any question for this introduction. No. Okay. Should I take the microphone? Yes, please. Hi. Thank you. All right, let's look. Okay, we've got the virtual box to work. Oh, am I unmuted? Can you hear me? Oh, good. We're all set. Okay, thanks Leandro. So I will show you a little bit of the code and we'll see if you can also replicate some of this. Who is able to also join me with the code in the machine? In the virtual machine? One, two. Two people. Okay, so for the rest I will just show the code and then in your own time you can have a look if it's also applicable to your projects. So this is the Jupyter Notebook that we will look at today. And then here we see the example of the interface, the online interface that we use to also publicly make it publicly available. And here we have the reference to, because much of this work is from Martijn's work that's also here, which will soon be publishing on this as well. So that's also interesting, I think. So first of all, we load the main packages that we need. I will reset the kernel so you can see how it's working. It should be working. Let me also clear the outputs. Is there anyone here who worked with SK Learn libraries before? No one. I think I know some people who did. That's okay. Okay, so you'll be recognizing some of this code here. So we will be loading the EU Map package, which is also, as Salerno mentioned, the main package that we, the current workflow that we use for the data that we provide. And we use some standard libraries, something to find our files in the system, and to load these files into the environment. So let's load these packages. Then I will define the tile that we use, so we provide two example tiles that you can try. And this, I just run this to be able to find the data on my drive. So here we have the file system, and I put the directory of the data. So that's this directory. Okay, so let's run that. Then I have some training data that is here. It's a geo package. So that's basically a lot of these points that are mapped that are we know, okay, this is this land cover type that is all put into this database. And then we will load it using the geo pandas package. So now this data is loaded into the variable called points. So here we see just a regular table with Yeah, all the information. So here we see for this, for this point, we know the class is 112. And then this is the value that the distance to the coast. This is the weighted distance to the coast, I think that Martin developed. But also if we look further into the data, then we see the land set values, and also some the names and the survey year. And because this is a year, this is a, yeah, this is your reference data. So we here see geometry. And the nice thing for the geo pandas package is that we can also see what what what reference system is used. So this is just the, this is a geo pandas data, data frame that has this, these functions to to find all this information. And actually see what my reference system is for these points. Because if I if I look at these values, it's not is not the regular coordinate so here you can see how how should I translate this if I want to plot it in another kind of software. And I want to get some information about my data. So for instance, so I just make a kind of a bar bar bar graph about some of my values. So here for instance I have the class name and the survey year, and then I want to see how many points do I have in my training data just to get an idea of what, what I have here. So I have a lot of coniferous forest points and more San Heathland and agriculture with natural vegetation. And here I see when these points were measured so I have most data in 2018. And this we can, you know, I generally I would do a more in depth look at these data but for the, for this presentation I will just go ahead with looking at the random forest. And then I will, I will focus on doing just the spatial and then the under a will later also include the spatial temporal so then we will see. I will be focusing all the years. Okay, so I will look at the spatial temporal data actually. So here I am also including some, some escalern functions. So this is actually to do. I think we use. Yeah. So this is for a bit later where we use the hyper parameter optimization, I'll come back to that. Then the random forest layer classifier that's one of the classifiers that's included in the escalern package. And there's many, many types of classification algorithms, and they're all kind of standardized so we try to also enable you to use all these different classifiers which makes it really versatile in how you can use it also on your spatial data so I think that's that's a very cool feature of this package. So the K for this to do, yeah, K, K amount of faults on your data to get a, an interesting evaluation metric. So here I will initiate the estimator the random forest estimator with five estimators. I'm going to select points just from this, the tile that has my tile ID that I have defined before, and I will use a five fold cross validation for the training. And then I define all the, all the variables that I want to use for my training. So in, in my data set, all the columns that we saw, I will put the prefixes for all these columns here. And then I will set up the, that I know which, which variables I'm using to train my classifier. So this is something that, that was written into this package so you can really be quite sure about what you train on and how you, and it's easily configurable. So I have my target column, that's our land use class. So I will load the land mapper class with all these configurations. So I have my points data for my data frame with all the training data. I have the variables that I want to use so it will be selected from this data. And then I have the target. So what should be trained on and then the estimator, the, yeah, the class that will handle the faults and another parameter where I just select classes that have enough training points. So I'm just going to look quickly at this one so it will select only the classes that have the highest amounts of, of training samples because for these ones we cannot reasonably believe. I think that we can make a good prediction because we don't have a lot of training points. Can I just explain it because we included this parameter because when you do a cross validation, you need. You want to have a look. Okay, so when you, when you use like this, but this parameter, it's important because when you do a cross validation. Mainly you will divide like imagine you have all these points you divide in five regular groups, but you need to have enough validation points in each of these groups so I don't know maybe you have just one or two points for some class. And this point will be present only in one of these folds in one of these regular groups and sometimes some algorithms that doesn't manage well with that. So, you have like, you need to provide it because it depends off the, the, the algorithm you can classify with less amount of samples. But for example, I know that SVM has problems with that. XGBoost also has problems with that. So there, there is a limitation with the way that the validation, the cross validation it's implemented considering this regular fault because you need to have all the possible classes in each of these folds to do a proper validation. Okay. Thanks. So, again, we, so we also focus on that it's usable in a production environment. So maybe sometimes you don't want all the, all the outputs to be printed to the, to the, to the terminal, or whatever and then you can also put, you can switch off all the outputs. So that's also useful feature I think. So let me see if I run this, I run this. I run this block of code. So I get some information about what the land map is doing. So it's removing some samples because it doesn't, it doesn't comply to this condition that I specified as they under just explained. You can just explain that this is actually the percentage right. The percentage percentage. Yes. So it's the percentage for each class. Yeah, just just back to that logic just because I think it's easy to this one. Yes, just, yeah. So it's, if they are less than 5% of the total of the samples they will not be selected. Yeah. So they, these are less than 5% of the total of all the samples that we have. So these are the original classes, the numbers and then they are remapped to this because some of the packages in as killer and they need us to have, they cannot handle these kinds of numbers to do classifications. So then we will train train this one. I'm putting this number so that we have the same selection of folds for our training. So we can have a bit similar results if you run it on your own machine. Then I can look at the accuracy. So here we can see all the accuracy metrics. So we get a 56.6% overall accuracy. And then for each individual class. So if you want to know which one it is precisely you can have a look here, how they are mapped. And we will also print a confusion metric metrics that will show us some in a bit more visually appealing way. So here I am just creating because the other names are in the, are in the data that we provided. So I'm just creating a dictionary to translate the number that's mapped to the actual name so it's still understandable to us. So here we can see the true classes and then how the model is predicting them. So for instance, let's see, sports and leisure facilities are confused with broadleafed forests sometimes. Or yeah, that's happening sometimes. So this, this you can use to get some information about how your model is performing and maybe where you need to see if you can do some feature engineering to increase your accuracy for specific classes. Let's see. So then you can also access this in in a tabled way. So what you expected the class to be for every prediction that we did. So then you can generate any type of accuracy metrics that you would like to do with your own functions. So after training, then we can save this model so we can use it in a production environment or something somewhere else, where we don't have to retrain it every time. So I'm just going to save it to this, this file. So we can find it here so the, it outputs where the location is. So I'll show you where it is. So this is actually my model now so I can actually just put it on in somewhere send it to someone and he can also try to predictions with the same thing using the EU map package. And now I'm just going to reset the kernel to see if I can actually load and do predictions again with the same, same model. So I have to reload all these packages because I just killed the whole instance. And then I'm going to define the parameters again, because everything is gone. And then here I'm going to load my model again. So this into this variable. So, we prepared all the data into these directories here that Leon would just talked about so some timeless layers and also the, I won't open it because it's many files so it will take some time. But there's a lot of lens at all the lens at bands that learn would just explained for every season for every year they're all in here. And for all the quantiles, but these two functions they will find all these the past for a specific year that I want to predict so now I'm just going to predict one year. So I will find all these layers here, I just print five of them so for one year I have. Let's see. I put them together so I can import them into into my class into the landmapper. So, so I have 86 variables for this. For this prediction that I need. So that's just, I'm just, I have all the past to all the variables that I need into the in this variable. And then I need to rename them because the original ones they didn't have a year there. So now I have in every name in every data that has a year component I have the year also in there. So I want to only get the 2018 data. So, yeah, replace the year with 2018. So I can rename all these files so my, so the model can find the right data to predict on. So it loads the right files to do the prediction. So here I put all these, all these parameters into the model line. I don't, I'm not interested in the output. We just looked at it. So we're not interested in in how the renaming works, because it's a long list of names that will be replaced so it's a bit. That's a good point. So let's see what happens if we show it makes more sense actually so you see what. So here we see that the original name that I had in my model for this variable it was this one so it has no year components, but I want to find this one where it says 2017. So that's what this function did so I make sure that the names of the files are right. So this. Yeah, it's a general thing in, in I think machine learning to, you know, you have a lot of preparation on your data so this is how we help handle the preparation of your data and keep it kind of comprehensible. So let me do that one more time without. So we have one output. I think so now it's doing a prediction. So took about one second. So here's my output. So I can plot that output with also one of the plotting functions that we provide. So here we see what the classification found based on our training data and the model. Let's see if you can find that one. If it makes sense to us. So here it is in on a map. So here we have the Netherlands we are now right here. So here we are we here. I think this is us, we are here. Right here. Okay, so let's see what what what is it. So we are value zero. Seven. Okay, so we're seven. So let's see what it is. If that makes sense. Pastures. Well, we tried this was our first try. Okay, so let's see what this this is zero. So that's more than Heathland. So this can help you kind of see, you know, what's happening with my model. You have a question. Just about the credit sample. Yeah, we have it. But actually we, we have like, we have like, you know, we have all these package relies on the functionalities of signkit learn inside of this signkit learn. And you'll have a validation strategy that's used like a crew by being to split the data. So you have this C.V. group. And column. So what actually what we do in the, in our production work, we did that. So all the tiles that I presented in the morning. So they, they are like one. And then we have the C.V. group. So you have this C.V. group. And then we have the tiles that I presented in the morning. So they, they are like one 30 kilometers by 30. So we use it like the tile ID as the group column. So to do like this special, this the cross validation, it will be like sort of spatial presentation that will use this ID. So you can avoid, for example, train a model with samples from one tile and validate with samples of the same time. So, and, but it's, it's just like, you need to change the CV to group and fold. So it's, it's a, it's implemented, it is implemented in the site kit learn. What we did was just like a integrated integrated to this functionality. So you can use this one, right? Yes. So you can use this one. So now I use here somewhere. Let's see. You just use the, yeah, because here we are just using, for example, the, it's just one pile. So we could split this point in like soup tiles and create this blocking structure. But for, for this example, probably it will work, but we need to really define this in the data frame. So that's a, that was a good question. Yeah, good question. Thanks. Let's see. Yeah, so in here, I think you can see, yeah, now we select on the tile ID, but you can then use it to do a spatial cross validation. And the next thing is that so all these, all these things you can use them within the package, right? So you can, now we use a K fold, but now we spoke about this group K fold, but you can apply different kinds of strategies here. Does that make sense? Yeah, great. Okay. The models. Let's see. I think. Where are they? Let me look further. This one. Yeah, so you have, because this, this, this libraries is, they create like this base class, this base estimator. So all the algorithms that are implemented here, they have kind of the same methods. So our package actually use all these methods training predict and things like that. So for example, you have like the SVM so if you replace the rental forest for SVM, you will have the same structure that we are showing here, but with a different model, the different learner. I think on top and bottom, you can see that there. So you can use like a linear regression. So we will use more learners, but it's fully compatible what we have here in the. Yeah. And I think you will show an example where we use multiple learners to do a ensemble machine learning. Okay, so we have now looked at this example so our first try we didn't do any hyper parameter optimization we just train a very simple model on our training data. And then now we will try and do some hyper parameter optimization and see if our results are actually better. Yeah, we already saw this one so this allows is also as a learn functionality that we will also use. Then the rest, this is kind of similar to what we did before. But here we define our grid search so we will run over the different parameters that we want to test. We want to test maximum depth and the maximum features and the number of estimators. So we want to test all these combinations of parameters and the rest and we use F one score that's also a standard scoring in in the S K learn package. But here we can also define our own. I think we will do it a bit later. We just input the estimator. And then the only thing we need to do here is just add the high parameter parameter and that will allow us to actually do this. And then we will create search. Let's see if we can do this. So we get the similar outputs. And then now we will train. We will actually train the model 90 times and see which one is best. So that might take a little bit of time. Let's see. What the computer is working on it. Some nice green bars. So it's still running. So now it's almost done. I think it's done. Yeah. So now we can see if the prediction actually better than before. So we have. What is it about a 6% increase, I think, in accuracy. So it's, it's, it's something. It's an increase. So that's good. Then we will do the same. We will, we can plot the. Yeah, the, the methods to do some invitation of our results. And we can look again at our results. We can save the model again and load it again. And I think, did we get the output file. Or not. No, okay, we skip the prediction. Okay, but we can do a prediction model. Well, let's have a look first at the probabilities and then I will, we can have a look at also these files. Now I'm using log loss score. So this is an example of a custom loss function that we make. We also use the escalator and implementation of this, adapted to our, our case. And we do the same thing here, but here we also tell our class that we want probabilities. So because we use this random force with a lot of different predictors we can see if they are corresponding to each other or not and if they are all pretty sure about the class then the probability is high and otherwise, the probability is low, or could be lower. So we make prediction again. And let's see if there's different results. We need to train it of course. So now we use a different loss function so we will. It's a bit different to compare it. So here we have now our accuracy metrics but then get towards the last metric that we defined for each class again. So that's, so we get from the model we get different types of accuracy metrics that we can all access through through these probability metrics variable. And then we can also print them. All right. And then we can see if this loss function actually gives us a better results. By looking at the, by looking at the prediction results. So now I'm applying the same function so to predict with the layers. But here I'm putting all the years. No. So now we're still looking at just 2018. So here's the same function to find the right data and to put all that together. And then we're loading it into the landmapper class and producing a prediction but now not just the class but also the probabilities for the different classes. And we also provide the plot roster function to actually have a look at the results. So, yeah, it's of course, very rough interpretation but if you look at this one compared to the other one. Let's see where it is. You see it's a bit cleaner. It's not sure if that's better but at least you can visually appreciate the difference in the result. So if it's better that's of course matter of further interpretation and looking at the data. But now we have a bit more more at hand to actually do these interpretations. So we can now also have a look at the, at the probabilities. So I think these are related with. So one is related with zero in this case. Yeah, okay. So I'm not trying to visualize the probabilities for the Morrison heat length that we just produced using the landmapper class. So, here we see the probabilities for this area. So again this is where we are now. Let's see. So this is the bargaining area this is bargaining town and here is the conference center. So now I can have a look where it's likely to have more in heat length. Yeah, let's look what's this. So this actually airports. So at least now we can see what what what areas have impact on our, on our predictions. And I should actually what I like to do is, because it's still the easiest way to to act to have compared these things to go to our viewer and see, you can also see the Lancet data over time. So that actually gives us a lot of information on what it is we actually see in our training data directly and then you can. Yeah, it's not. You can probably also load it in GIS but this one is like really, I think really user friendly to go to different years. We have to. Okay. This one. Yeah, yeah. That's it. So here we can have a look again. And now we have layer coniferous forest. So let's have a look. If this is corresponding to what we found. So here we are. It's a tricky class, because it's very much like I think many other classes. But hopefully our model also picks up these other classes and actually produce higher probabilities for these so that even though we see high probability here, maybe other classes still have higher probabilities there. So let's have a look. We now created our own prediction here. Let's see. Let's stay close to where we know where we are. So this patch for instance. Yeah, that's different. So now I'm observing that our predictor kind of finds a patch of heat land here or at least a higher probability than that we have found in the prediction codes that we used for the Yeah, that we used for this. So that's interesting I think you know this is. Yeah, just some interpretation of the results that you can do and I think all these tools they really help you with. Yeah, visually interpreting also your results and see what what you find in your training data. So that's very interesting. Let's see where we're, and you can do this for all these classes so this class five for instance that's going to first forest. So this makes a lot of sense if I look at this because if you are here, Wageningen then this is the veil with that's the biggest force in the Netherlands. So for me, I think okay this is at least there is quite some some truth in this prediction because I know this is forced area. Of course, there are many much more research to be done to confirm this. And then I think the under all you want to show the people the ensemble machine learning. Yeah. Yeah, okay. Thank you. So I'm not sure any questions about this part. Please. No. Okay. Yeah, so. And now we have like the, the probability results. And I would like just to. So, please show how to execute it. And I would like to explain this report here. And the metrics that we generated so. Here you can see that this is, you can see this report because Chris instantiated the landmapper with the verbals through so even for example for for us that we are developing this this API and we are using like on a daily base. It's really important to follow the process so follow how what it part of the, the, how which part of the workflow it's executing. So these verbals it will like explain everything that happens in the landmapper. So, the main, the main points here it's really, you can use all the class that are available in the signkit learn. So, random forest for learners but you can also use SVM you can use a Keras classifier that it's a implementation compatible with signkit learn. So I will use it, but you also can use it individually. And you can use different types of validation strategies. So here we are just using mainly like cross validation so it's, it's, there are a lot of discussion if the cross validation it's, it's the right way to evaluate it. But what we decided when we were implementing the library is inside of the library will mainly use cross validation. So if you want to validate your result you can do it outside so it's just a matter of have independent points and do the predictions using like the predict points method. So, Chris presented the, the interface here so you can see that you have these predict points. So if you have like independent points you can manage it outside. So, and, for example, you here we are using these Lucas points, combined with Korean Korean land cover, and we are creating our data set. And for the production work actually we use it almost 6 million of points so it's much more points that we are presenting here in this training because it here it's just a matter of how we demonstrate these functionalities. But, for example, we have the model train it now our production model like to produce the maps that are available now. And we can conduct, for example, I independent validation strategy and we can spread points all over the Europe and expected it by visual interpretation or even go to the field if we have money and the context of the project, collect the actual class of it and just use these points to do another prediction so it's a completely independent approach that we, it's not really related with the training and how we are implementing these inside of the model inside of the land map. So, you have this, this possibility. But what it's important here is, we are using the cross validation, because it's a kind of, it's a kind of metric that allows you to evaluate your training process, how you are capturing the variability in the class how your features are really like correlated with the, the expected variable and you can use this, like a reliable way to see if the model it's doing a good job, but to provide like a real validation of your result. And you have like independent data set and it's better provide like a proper sampling design so spread points considering like all the assumptions of the statistics and like a random sign or stratified design to do it but of course, sometimes you are limited, you don't have like budget enough to collect points to produce new data. So you need to work just for example with cross validation but in the literature now you have this kind of discussion and it's, and I personally I don't believe that a cross validation it's able to provide like a real and a proper validation for your prediction so it's a nice way to understand if your model it's doing a good job. It's a nice way if you, to understand if you are optimizing it, but to provide the real validation it's like independent data set so it's, it's, it's what we presented in the beginning of the presentation so you actually you need to have no new data so and, and you can try to split it, but for for the context of the land cover it's it's, it's how it's the state of the art, sampling design to provide a better validation and a proper validation for your product. So, you, you have it here, not all these aspects but you have the possibility to use what we train it now to predict new points, not just the, the, the, the, raster raster files. But if you back here. So you can see that considering this cross validation strategy that we are using here it's a five fold cross validation, the landmapper, it's actually dual optimization approach to find what are the best hyper parameters. And it's find the best, it's something like vague. So basically what it's doing, it's calculating some currents metric. And here you can see that Chris explained it. So you have here it's a F1 score weighted and it needs to be weighted because you have, it's a multi-class classification. But for example, if you click here, you can see that all those metrics are really available in the signkit learn. So we could optimize also by the currency or by the precision and by the recall that are actually in the context of remote sensing are like user and producer currency. So it's possible do it. But so it's really like it's a modular approach. So you have like validation strategy defined, you have which parameters that you want to test. So all the parameters that I put here, it will be a combination of all these parameters. And you have different strategies also. So I always prefer to find the parameters because it's sometimes you really need to think about what not all hyper parameters needs to be optimized. And so, and it depends of each algorithm. So define these values actually force you to understand a bit more about these learners, about these models and select actually what are like the best, not the best, but the proper parameters that you need to optimize. So in here, we are optimizing the number of trees. So it's something that it's a common standard in the most part of machine learn applied to land cover classification and other domains. But if you provide like, I don't know, 60, 80 trees, so you should have a good result. So considering like dominant class, hard class. So it's a kind of standard value. But of course, if you put more trees, you will provide, you will have a higher computation time because each tree needs to be computed, need to be trained, need to be predicted. So, and this number, it's actually, it's really high correlated with the performance of the model. So for example, if you have 40 trees and you feel you have 40 course, that's not our case here, definitely, we just have four. But if you have like 40 course, each of these trees will run in parallel. So the process will be way faster. So it's this hyperparameter optimization, you want to optimize to find the best model, but there is a plateau. So you need to be aware that sometimes you are just adding computational time and it will increase, I don't know, 0.5% in your records. But for some applications, maybe this 0.5, it's really important. So if we are doing like with, for example, medical applications, it's life. So you need to optimize at most, but it depends off the way and the domain and the application that you are trying to do. So you have this, for example, flexibility here and even for land cover and for geo-spatial classification, maybe you need to really improve the accuracy at most and process more with a high computational cost. So it really depends. And here you have like these other three parameters that are actually the, what is the maximum depth of the each tree? So if you think that these three will like be trained like you will split the data set on each node of these three and this maximum depth will be actually, the default parameter here, it's non. So it will grows till like separate all the samples in the training set. But sometimes you, if you reduce this depth, you will have like a shorter trees and maybe it could help for the probabilities because you will have like more fuzzy output. And if you combine all these trees, maybe it will help. But it depends off your model and the way that you are training. Not the way that you are training, but your training data set. And the maximum feature, it's actually how much features it will be analyzed for each split. So for example, if you put here 0.5, we have 86 features. So it will use half of it to define the best variable to separate the training samples. So, and for example, sometimes I use it like one here. It's totally random. So the random forest will be trained like for each tree and each node, each part of these three, it will select randomly one variable and it will try like split. And maybe this variable, it's meaningless. So maybe this variable, it's not really performing well to separate your point samples, but it will create this node past to another level and select other variables. So maybe you could have like a really deeper trees, but it could help. So it's difficult to predict it. So you need to really think in some parameters, think about what are your computational capacity and think in how you will do the predictions. Because I don't know, maybe we can have a random forest with 1,000 trees, but how we will run it. We have a server to do it or maybe it could produce a very long computation time. So you need to balance all this to define these grids or CV. So it's why I really like to not use them, there is one strategy that you just put some random value between like one range. But I think it's nice to, if you think about it and really try to put some values that are specific for your domain, for your problem and for your application. So mainly here, what we are seeing is all the so you can see that there are 18 candidates. So it's mainly three multiplied by three. And so yes, three, three whites should be less. So 18, yeah, three. Yeah, three by three, nine. Yeah, multiplied by two, 18. So actually, if you think in all these combinations, you have 18 combinations of hyperparameters. So it grows fast. So I don't know if I put like more two parameters here in each of these options, it will be more than 100 combinations and you can quickly blows the process. Not here in our case because it's a small dataset, but mainly for all these 18 candidates, we are using a five-fold cross validation strategy. And so actually we are training 90 models. So, and here you can see the result of the F1 score, weighted F1 score for each of these combinations. And here we are optimizing it by the higher value. So, and the best parameter is this one. So maybe we could optimize a bit more, but for this dataset and for the random forest, we can put more trees if that was the, maybe that could help, but maybe not too much. So, yeah, I think this, yeah, I think here we stop for a break. And I will back for the second part and we can analyze better the probabilities and predict other years. So, and there is, I don't know if there is questions, there are questions. Nothing to chat. So, okay. So let's have a break and we back in half an hour, right?
 Let's back to the training sessions.  So this in this afternoon, we will use two slots to present about the spatial temporal  machine learn in Python and how we implemented it in the context of the open data science  Europe.  So the presentation will be, we will start with me and later Chris will join to do some  code demonstrations as we are doing for all the sessions.  And we'll have a break and after that we will back and we will discuss more some about the  ensemble machine learning.  So and the, to the approach that we implemented now in the EU map.  So before to go to the code, I put some slides.  So I prepared these slides to provide some conceptual background about machine learn.  So how we it's important to provide like a common understanding of what we are doing  in a general way, because of course machine learning can be used by different applications.  It's kind of buzzword now.  So everybody's talking about it.  So considering this general introduction, I will explain how to do it specifically in  using geospatial data.  So what are the, what is the approach?  What are the data?  How we can implement a processing workflow to do it?  And specifically on that, I will show two demos, two real use applications that classify  a land cover.  One, it's purely spatial.  So that I implemented in my PhD.  And now it's spatial temporal and I implemented, we are implementing it in the open data science  Europe.  So I see it as a kind of evolution between this purely spatial to spatial temporal.  So there are some important aspects I will explain about it.  And after this introduction, we will have a hands on like a demonstration and we will  actually we will execute the code using mainly the EU map package.  So there is a class that we've created, landmapper, where you can implement all these machine learning  processing specifically for geospatial data.  So let's start.  So I like to, I would like to start with the definition.  So what is machine learning, and if you think in a nutshell, it's essentially a subfield  of artificial intelligence.  So it's not the same thing.  You have more subfields inside of artificial intelligence, like robotics and fuzz logic.  So machine learning, it's a subfield inside of it.  But most important, it use techniques to learn from the data and make accurate, accurate  predictions.  So the whole point to use machine learning is train over like some specific data and  do predictions in new data.  In the field, we call it like a generalization.  So you receive new data that the model didn't see and you produce good predictions.  And there are several ways to implement it, different algorithms, but in essence, it is.  And other important part is without being explicit programed.  So if you think what it's happening inside of machine learning model, you can code like  hard code, all these rules.  So if you think like a developer, I could get an image and do a lot of rules myself to  try to identify, for example, where is water and work with the data, try to find some values.  And I can code that.  But if you think in the way that machine learn do it, it implements it for you.  So it will create all these parameters, look in the data and try analyzing the data and  try to establish it like automatically.  So without being explicit program it.  So if you think in the artificial intelligence field, you have other subfields, so machine  learning is one of these subfields.  And inside of machine learning, you have like other buzzword today, like deep learning.  So here we are, we will not talk about deep learning.  So we need to, we could have other session about it, but we will use TensorFlow.  So it's not every time that you use TensorFlow that you implement deep learning.  But and today all these concepts are kind of mixed.  So for example, in artificial intelligence, you have like a natural processing language  subfield.  And today you have pretty good algorithms and really accurate algorithms in deep learn  to deal with that.  So some of these subfields are really changing too much.  And deep learning, it's a really hot field in the research.  And there are so many algorithms and really nice implementations of it.  But mainly you, when we are talking about deep learning, we are talking like processing  graphical card interface, processing unit like GPU, convolution neural networks and  all, it's more related with it.  We have like a recurrent neural networks also.  But here at the end of the training session, we will use a regular artificial neural network  and it will run on forest and with the gradate and boosted trees.  So in this session, we are actually inside of the, just of the machine learning, not  actual deep learning.  So now if you think in the machine learning, you have mainly three like categories about  algorithms.  So we have the supervised, the unsupervised and the reinforcement.  So the supervised, it's mainly when you provide like training data and using this training  data, you already provide like the answer.  So you need to have like raw data can be image can be text can be a table.  It doesn't matter.  But you need to label this data.  So what it means, it means that you will look this data.  So some human being will look this data and it will create like what is the expected result  for this data.  So if you think like in an image, we need to provide for the satellite images, where  are the class that we want to map or the values that we want to map.  So for example, you could provide like coordinates points, specifically on top of the image showing  that this is a patchery, this is a conifer forest and things like that.  Or you could provide polygons also.  But you need to provide it.  So you need to have a data with label, so label data to do the training and the machine  learn will try to match and to analyze all this data to provide like expected results  and you can compare with that.  So here mainly we will use it.  You have the unsupervised learner.  It's more like machine learning.  So you don't need like training data and you can work with the raw data and try to find  some like hidden structure on it.  So for example, for the spatial classification, you could try analyze purely the values of  that image and try to aggregate values that are similar.  So if you think maybe a pixel in a forest, it's in specific part of the image, it's close  and or it's related has values related with pixels on other part that are forest also.  So it's possible.  But sometimes the most part of the times actually they are not really accurate.  So the accuracy, it's better if you have the label data.  But it's there, there are researching, there is a lot of research in this field also and  you have the reinforcement learner.  So machine learn this reinforce you learn by mistake.  So you have like actually an agent that try to do something and you have some way to measure  what to analyze if that action produce or not the expected result.  And it will learn considering the response of different attempts.  So I'm not familiar with this approach and you again, you can combine it so you can have  like a supervised learner and you can after that you can have like a reinforcement learn.  So but it's important to be aware of these these fields and these concepts.  And here, mainly we will work with the supervised machine learn a specific classification algorithm.  So we are not doing regression.  And if you think in a classification, the difference between classification regression  mainly we are producing like factor values or like integer values or class values.  And in the regression you are the model will predict like a continuum value.  So it's more like to predict some variable.  So in our case, the land cover, it's actually like a class, we have the same exactly the  same value.  It's an integer, but we have the same value for different pixels of the images.  But for example, if you think if we we could train a model to predict the precipitation,  for example, and to predict precipitation, the output will be like a continuous value.  Each pixel could have a some level of the precipitation and the this is more like a  continual value.  And if you need to use a regression to implement it.  So and the other part it's it's about it the program.  So inside of the the model, you will have like what we call like the parameters.  So the parameters are actually the how the model will implement all these rules, all the  training rules to split and to analyze the data and predict the expected result.  So to predict if that pixel, it's a class, I'm talking about image, but for example,  you can do the same thing for text.  So now in nowadays, for example, the Google translate do it.  So it analyzed some of the whole structure of the text and predict the output as a new  in another language.  And it's mainly it's the same concept.  So you have the input data set, it's actually text, and you are predicting and the Google  translate predicts it as a as a text, but in other language.  So and if you think in the way that it was implemented, any anybody created like some  anyone created some hide and rules inside of it, the model learned it considering millions  of training data in the case of the Google translate, but you can have it for all the  for different kinds of applications.  And one important concept is when we are talking about the model, the parameters are like.  We train the parameters.  So if you think in the random forest, it's basically a decision tree.  So you will analyze the data and this data could have like eight, 10 features, 10 variables.  In our case, it's 10 images.  We will use 84, but it could be like any number of covariates and features and analyze this  picture, this this features and this pixels.  The model will train like a tree and each part of these three, it's a decision tree,  it will like try to define to divide this data to classify in pasturing in conifers  forest.  So you the way that this decision tree is created, this is the like specifically the  rules.  These are the parameters.  The hyper parameters are mainly the they are related with how you control the learning  process.  So for example, if we use 100 trees or 10 trees for for the random forest, this is the  hyper parameter.  So number of trees is a hyper parameter inside of each of these trees.  We have parameters that will analyze the data and it will split it and it will generate  the output.  So this is the the parameters of each learner and different algorithms, they have different  parameters.  But mainly what you need to remember is the parameter, it's actually they are learned  from the data.  So and it's it's it's automatically derived considering it's a data driven process.  So you don't need code anything inside of it.  You just need to code outside to control the process.  So this is a expected machine learn supervised machine learn approach.  So we have some raw data using this raw data, you can have features.  Again, this could be anything tables, text, images doesn't matter.  And you have labels labels are like, what are they?  What is the expected result?  Considering this raw data.  And of course, the raw data, you can derive just some features, you can have like a bunch  of raw data, but just part of it you are using to do machine learning.  And you have like the model training and the model evaluation.  And with a training model.  So after the training, you have like a model, a training model, you can save it, you can  load it.  But mainly you can use this training model to do prediction over new data.  So that all that effort creating the labels and create the expected result.  And there it needs to be currently actually it needs to be done by someone.  So human human being and me.  And you can use it to predict and to generate predictions.  So this is the expected workflow, but for the real world applications, we have more  something like that.  So it's not so beautiful, but you have the when we are talking about real applications,  you have like a way more complex, for example, data and raw data than this benchmark data  sets.  And when I'm talking about benchmark data sets, I'm saying like, if you download, for  example, the TensorFlow, you have like some digit numbers, like images with digit numbers,  and they are really organized and pretty and ready to go.  So you just need to get this data and do some predictions and you already have the labels.  But the most part of real applications, they don't have the data really organized like  that.  You need to prepare the input data set, you need to prepare the labels, you need to expect  to prepare the expected result.  And you need to work all with all the modeling part.  But at the end, you need to do predictions.  So otherwise, you are not interesting, just create a model and use this model to just  to predicting just a portion of the data.  What is the end goal considering a real use application here?  And for example, in our case, the end goal, it's really mapping the length over, map the  length over classification and the continent of Europe.  So we did all this effort, but at the end, we need to predict it.  So we need to get this, this model and really scale it up and predict all the pixels of  Europe.  So there is a way more complexity than just like a simple workflow.  But mainly, if I could like put the two most important aspects, you need to prepare the  data before to really start doing modeling, starting to machine learn.  And you need to be aware what you plan to do with the final model.  So what will be your prediction strategy?  So you can put it in a, I don't know, in a service and the users can access the servers  as for example, we have for the Google Translate, it's a service for a machine learning model.  And everybody can go there and just put some text and it will translate.  So considering a real use application, you need to be aware of how you will serve your  predictions.  So you can do all the predictions in-house as we are doing, generate the maps and put  it available to the people who use the maps or we can really serve the model and to the  other users really access and do new predictions.  So there are several possibilities on it and it's important to be aware of these possibilities.  So but let's back to our simple example and we can expand it for the geo-spatial data.  So now if we specifically explain it in the context of the geo-spatial data, this is more  or less the pretty way I say it.  So you have like the raw earth observation data.  So this earth observation data as I was explaining the morning, so you can have like the Landsat  data, Sentinel data, there are a lot of satellite images available, but you need to download  it, you need to organize, you need to remove the clouds, you need to do several pre-processing  steps and to produce analysis ready earth observation data.  So and using this analysis ready earth observation data, you actually need to provide now the  labels.  So what you want to classify in this image.  So you want to classify like soil organic carbon, you want to classify land cover, you  want to classify, I don't know, just one class, pastry maybe, or you want to classify water.  So you need to provide the labels because the performance of the most, the better models  that are available now, they work with supervised classification.  So you need to provide the samples, the training samples.  And the training samples, it can be like points, lines, polygons, it doesn't matter.  A sample, it's just like a vector that you will overlay with the data, get the pixels  and put inside of the machine learning.  So here in the training session, we will show the like an overlay of data, but in the U-Map,  so look at, did talk about it in U-Map, you have like the functions to, I don't know,  get one point that samples that you collected in field, for example, and overlay these points  over like raster data, different raster layers.  So we are, we, in these training sessions, we already did it, so we will work just with  the overlay of data, point data, that for each point, we extracted all the pixel values.  So and considering it, we will train models, so we need to train a model, we need to evaluate  this model, so there are some strategies to do it.  And after that, we will have like a trained model, and when I say like trained model,  it's a ready to go model, so it's a model, it's a file, it's a model that it will be  able to do predictions without do the training again.  So we will show it for you in the code.  So mainly when you do the training, you have like a model, and this model, he has like,  he knows how to process new data and predict like the land cover.  So and he did learn considering just the samples, but you can use to do predictions in different  parts of your data, of course.  So there are some limitations with that, but then go with, is you really, for example,  train a model to do the prediction in the whole year.  So as we are doing here, and we will, and you can load this model and you can skip all  this training process and just predict.  Yeah.  Yeah.  If it's straight going, that's the idea that you will be able to process analysis, right?  Rather than the route.  No, that's a good question.  Yeah.  So actually, you need this rule, this arrow here, you need to read from the analysis ready  data.  So because, yeah, I will fix it in these slides.  Thank you for noticing it because this, the training, it's really sensitive.  So if you, if you change a bit the pixels, you will change completely the predictions.  So it's, it's, there are some techniques to really try to put, like you can simulate  noise in the analysis data and to try to predict raw data, but it's, it's more, it's a different  type of research.  And here we are predicting analysis ready data.  So this is one of one example.  So I did it in the, my PhD.  So for, I worked in Brazil using the Landsat data.  So mainly I, we had like more than 30,000 points for each year.  And these points were visually interpreted by three specialists and they classified as  pastry, forest.  So and they are, you can see the points here.  They are really spread all over the country.  So this is a, we, and in my, in my PhD, I was part of the GIS and remote sense laboratory  in Brazil.  And actually we generated this data from scratch.  So here we actually develop all this.  We develop a sampling design.  We spread the points all over the country and we develop a solution to help the specialists  to check the pixel values and see if it's pastry, if it's crop and things like that.  So in this context, it's, it's, it's not like it's really specific, but the, what it's  interesting here, we have samples for the whole country and for the, all the years.  And that's, it's not the main part of the applications.  So, but even have, so considering that we have points for all the years and for all  the country, you can see here, this little square, little squares here are like, those  are the Landsat scenes.  So we have 308 tiles, Landsat scenes to cover the entire country.  And we have 34 years.  And what we did here were really, we, we, we, we trained one model for each scene.  So if you think in the machine learned way, the training process occurred by each Landsat  scene and by each year.  So at the end, we trained more than 2000, almost 13,000 random forest models.  And it, and we mapped individually all the years.  So, but if you think we have all this data, so it would be possible training one single  model.  And what I mean with that, I mean, I can get all the points from all the years and for  the whole country and put all these points in a single machine learning model and train  it.  So it will be millions of points.  And but maybe it could produce a better result.  And but they're mainly there are two problems with that.  We need to have the capacity, software and hard to do it.  So it's not easy training a machine learning with millions of points.  And we need to have our monizer data.  And if you remember, my first in the morning, I explained that if you work with just with  the raw data with the lens, we have three sensors.  These sensors are different.  So one image of 285 will not have the same pixel values, the same reflectance values  from like an image of 2020, accurate by Landsat eight.  So you need to work with harmonize it, the Earth observation data.  So and they are true products at least probably there are more.  But the main products to do it, it's the glad Landsat R&D.  I talked about it during the morning and there is this other product that combines landsat  and sent in out shoe.  So in produce like one harmonized data cube, but it's just with Landsat eight.  So we will have just since 2013, but I think they process it just from 2015.  So this is mainly this is mandatory to try to create one single model for all this to  classify for example, multiple years and different parts of Europe.  So in the open data science Europe, we are doing it mainly.  So we took all the Lucas points and that it's a basically a well-known field survey here  in Europe.  And we combine it with some generated points from the old from from the other land cover  maps Korean land cover classification.  And mainly we are using all these points to train one single model, but it's important  to emphasize that we have the capacity to do it.  And we also have like a harmonized earth observation data as the glad Landsat R&D.  And without gaps and all these parts that I was explained during the morning.  So at the end, we are predicting dominant class probabilities and uncertainty.  So in the end of this training session parts to you will produce it also in the Python  and notebook.  But we are also using more than one learner.  So random forest, it's already a sample learners because inside of a random forest, we have  several trees.  So decision trees and it's why they call it like a forest.  It's multiple trees.  But we are also using the gradient boosted trees and the artificial neural network.  So we will show how to do it in Python for the tiles that we prepared.  And yeah, now it's time to go to the code and start to really do all classifying the  land cover, considering the data that we prepared.  So just to overview of what we will do mainly, we will train a regular random forest, pretty  straightforward without any type of hyperparameter optimization.  It will be just a purely random forest and we will predict it to see how it works and  to understand how we map it's doing it internally.  Later the second example will be the random forest with hyperparameter optimization.  So you can test different combinations of hyperparameters to really predict it and benchmark  all these different parameters and find what is the best one considering some evaluation  metric.  And the second part, we will train the random forest with probabilities.  So if you think in the random forest, you have multiple trees.  So each tree will produce like an output and you can use this output to derive like a kind  of a probability.  So and this probability, it will, it changes completely the output of the algorithm.  So we will see it, but instead to just have a hard class, a dominant class, so it's pastury,  forest, you can have like multiple probabilities and these probabilities will show, for example,  maybe there are some, there is an area where you have like a not a high probability, you  can have a high probability just for one class.  So you have sure that the model is pretty sure that this is, for example, coniferal forest,  but you can have like other pixel where the coniferal forest is 60% and the broadleaf  forest is 40%, 30%.  So there is some confusion there.  And if you think in the image, man, it kind of makes sense because you had some limitations  with the data that we are working.  It's not possible really get one pixel and have around 100% sure that inside of that  pixel we have just one single length over class because we can have in a transition zone.  So there are several complications of it.  And at the end, we will expand this and put more learners, so more three learners and  explain how we are deriving the uncertainty and how we can improve this result.  And this is actually our current workflow.  The code that we will show here is actually the same code that we are running in our servers  to produce the length over map available in the open data science Europe.  So now I will pass the floor to Chris.  I don't know if there is any question for this introduction.  No.  Okay.  Should I take the microphone?  Yes, please.  Hi.  Thank you.  All right, let's look.  Okay, we've got the virtual box to work.  Oh, am I unmuted?  Can you hear me?  Oh, good.  We're all set.  Okay, thanks Leandro.  So I will show you a little bit of the code and we'll see if you can also replicate some  of this.  Who is able to also join me with the code in the machine?  In the virtual machine?  One, two.  Two people.  Okay, so for the rest I will just show the code and then in your own time you can have  a look if it's also applicable to your projects.  So this is the Jupyter Notebook that we will look at today.  And then here we see the example of the interface, the online interface that we use to also  publicly make it publicly available.  And here we have the reference to, because much of this work is from Martijn's work  that's also here, which will soon be publishing on this as well.  So that's also interesting, I think.  So first of all, we load the main packages that we need.  I will reset the kernel so you can see how it's working.  It should be working.  Let me also clear the outputs.  Is there anyone here who worked with SK Learn libraries before?  No one.  I think I know some people who did.  That's okay.  Okay, so you'll be recognizing some of this code here.  So we will be loading the EU Map package, which is also, as Salerno mentioned, the main  package that we, the current workflow that we use for the data that we provide.  And we use some standard libraries, something to find our files in the system, and to load  these files into the environment.  So let's load these packages.  Then I will define the tile that we use, so we provide two example tiles that you can try.  And this, I just run this to be able to find the data on my drive.  So here we have the file system, and I put the directory of the data.  So that's this directory.  Okay, so let's run that.  Then I have some training data that is here.  It's a geo package.  So that's basically a lot of these points that are mapped that are we know, okay, this is this land cover type that is all put into this database.  And then we will load it using the geo pandas package.  So now this data is loaded into the variable called points. So here we see just a regular table with  Yeah, all the information. So here we see for this, for this point, we know the class is 112. And then this is the value that the  distance to the coast. This is the weighted distance to the coast, I think that Martin developed.  But also if we look further into the data, then we see the land set values, and also some the names and the survey year.  And because this is a year, this is a, yeah, this is your reference data. So we here see geometry.  And the nice thing for the geo pandas package is that we can also see what what what reference system is used.  So this is just the, this is a geo pandas data, data frame that has this, these functions to to find all this information.  And actually see what my reference system is for these points. Because if I if I look at these values, it's not is not the regular coordinate so  here you can see how how should I translate this if I want to plot it in another kind of software.  And I want to get some information about my data. So for instance, so I just make a kind of a bar bar bar graph about some of my values.  So here for instance I have the class name and the survey year, and then I want to see how many points do I have in my training data just to get an idea of what, what I have here.  So I have a lot of coniferous forest points and more San Heathland and agriculture with natural vegetation.  And here I see when these points were measured so I have most data in 2018.  And this we can, you know, I generally I would do a more in depth look at these data but for the, for this presentation I will just go ahead with looking at the random forest.  And then I will, I will focus on doing just the spatial and then the under a will later also include the spatial temporal so then we will see.  I will be focusing all the years. Okay, so I will look at the spatial temporal data actually.  So here I am also including some, some escalern functions. So this is actually to do.  I think we use. Yeah. So this is for a bit later where we use the hyper parameter optimization, I'll come back to that. Then the random forest layer classifier that's one of the classifiers that's included in the escalern package.  And there's many, many types of classification algorithms, and they're all kind of standardized so we try to also enable you to use all these different classifiers which makes it really versatile in how you can use it also on your spatial data so I think that's that's a very cool feature of this package.  So the K for this to do, yeah, K, K amount of faults on your data to get a, an interesting evaluation metric.  So here I will initiate the estimator the random forest estimator with five estimators.  I'm going to select points just from this, the tile that has my tile ID that I have defined before, and I will use a five fold cross validation for the training.  And then I define all the, all the variables that I want to use for my training.  So in, in my data set, all the columns that we saw, I will put the prefixes for all these columns here.  And then I will set up the, that I know which, which variables I'm using to train my classifier.  So this is something that, that was written into this package so you can really be quite sure about what you train on and how you, and it's easily configurable.  So I have my target column, that's our land use class. So I will load the land mapper class with all these configurations.  So I have my points data for my data frame with all the training data.  I have the variables that I want to use so it will be selected from this data.  And then I have the target. So what should be trained on and then the estimator, the, yeah, the class that will handle the faults and another parameter where I just select classes that have enough training points.  So I'm just going to look quickly at this one so it will select only the classes that have the highest amounts of, of training samples because for these ones we cannot reasonably believe.  I think that we can make a good prediction because we don't have a lot of training points.  Can I just explain it because we included this parameter because when you do a cross validation, you need.  You want to have a look.  Okay, so when you, when you use like this, but this parameter, it's important because when you do a cross validation.  Mainly you will divide like imagine you have all these points you divide in five regular groups, but you need to have enough validation points in each of these groups so I don't know maybe you have just one or two points for some class.  And this point will be present only in one of these folds in one of these regular groups and sometimes some algorithms that doesn't manage well with that.  So, you have like, you need to provide it because it depends off the, the, the algorithm you can classify with less amount of samples.  But for example, I know that SVM has problems with that. XGBoost also has problems with that. So there, there is a limitation with the way that the validation, the cross validation it's implemented considering this regular fault because you need to have all the possible classes in each of these  folds to do a proper validation.  Okay.  Thanks.  So, again, we, so we also focus on that it's usable in a production environment. So maybe sometimes you don't want all the, all the outputs to be printed to the, to the, to the terminal, or whatever and then you can also put, you can switch off all the outputs.  So that's also useful feature I think.  So let me see if I run this, I run this.  I run this block of code.  So I get some information about what the land map is doing.  So it's removing some samples because it doesn't, it doesn't comply to this condition that I specified as they under just explained.  You can just explain that this is actually the percentage right.  The percentage percentage. Yes.  So it's the percentage for each class.  Yeah, just just back to that logic just because I think it's easy to this one. Yes, just, yeah.  So it's, if they are less than 5% of the total of the samples they will not be selected. Yeah.  So they, these are less than 5% of the total of all the samples that we have.  So these are the original classes, the numbers and then they are remapped to this because some of the packages in as killer and they need us to have, they cannot handle these kinds of numbers to do classifications.  So then we will train train this one.  I'm putting this number so that we have the same selection of folds for our training. So we can have a bit similar results if you run it on your own machine.  Then I can look at the accuracy. So here we can see all the accuracy metrics. So we get a 56.6% overall accuracy.  And then for each individual class. So if you want to know which one it is precisely you can have a look here, how they are mapped.  And we will also print a confusion metric metrics that will show us some in a bit more visually appealing way.  So here I am just creating because the other names are in the, are in the data that we provided. So I'm just creating a dictionary to translate the number that's mapped to the actual name so it's still understandable to us.  So here we can see the true classes and then how the model is predicting them. So for instance, let's see, sports and leisure facilities are confused with broadleafed forests sometimes.  Or yeah, that's happening sometimes. So this, this you can use to get some information about how your model is performing and maybe where you need to see if you can do some feature engineering to increase your accuracy for specific classes.  Let's see. So then you can also access this in in a tabled way. So what you expected the class to be for every prediction that we did.  So then you can generate any type of accuracy metrics that you would like to do with your own functions.  So after training, then we can save this model so we can use it in a production environment or something somewhere else, where we don't have to retrain it every time.  So I'm just going to save it to this, this file.  So we can find it here so the, it outputs where the location is.  So I'll show you where it is.  So this is actually my model now so I can actually just put it on in somewhere send it to someone and he can also try to predictions with the same thing using the EU map package.  And now I'm just going to reset the kernel to see if I can actually load and do predictions again with the same, same model.  So I have to reload all these packages because I just killed the whole instance.  And then I'm going to define the parameters again, because everything is gone.  And then here I'm going to load my model again.  So this into this variable.  So, we prepared all the data into these directories here that Leon would just talked about so some timeless layers and also the, I won't open it because it's many files so it will take some time.  But there's a lot of lens at all the lens at bands that learn would just explained for every season for every year they're all in here. And for all the quantiles, but these two functions they will find all these the past for a specific year that I want to predict so now I'm just going to  predict one year. So I will find all these layers here, I just print five of them so for one year I have.  Let's see.  I put them together so I can import them into into my class into the landmapper.  So,  so I have 86 variables for this.  For this prediction that I need. So that's just, I'm just, I have all the past to all the variables that I need into the in this variable.  And then I need to rename them because the original ones they didn't have a year there.  So now I have in every name in every data that has a year component I have the year also in there. So I want to only get the 2018 data.  So, yeah, replace the year with 2018. So I can rename all these files so my, so the model can find the right data to predict on.  So it loads the right files to do the prediction.  So here I put all these, all these parameters into the model line. I don't, I'm not interested in the output. We just looked at it.  So we're not interested in in how the renaming works, because it's a long list of names that will be replaced so it's a bit.  That's a good point. So let's see what happens if we show it makes more sense actually so you see what.  So here we see that the original name that I had in my model for this variable it was this one so it has no year components, but I want to find this one where it says 2017.  So that's what this function did so I make sure that the names of the files are right.  So this. Yeah, it's a general thing in, in I think machine learning to, you know, you have a lot of preparation on your data so this is how we help handle the preparation of your data and keep it kind of comprehensible.  So let me do that one more time without.  So we have one output.  I think so now it's doing a prediction.  So took about one second. So here's my output.  So I can plot that output with also one of the plotting functions that we provide. So here we see what the classification found based on our training data and the model.  Let's see if you can find that one.  If it makes sense to us.  So here it is in on a map. So here we have the Netherlands we are now right here.  So here we are we here.  I think  this is us, we are here.  Right here.  Okay, so let's see what what what is it.  So we are  value zero.  Seven.  Okay, so we're seven.  So let's see what it is.  If that makes sense.  Pastures.  Well, we tried this was our first try. Okay, so let's see what this this is zero.  So that's more than Heathland.  So this can help you kind of see, you know, what's happening with my model.  You have a question.  Just about the credit sample.  Yeah, we have it.  But actually we, we have like,  we have like,  you know,  we have all these package relies on the functionalities of signkit learn inside of this signkit learn.  And you'll have a validation strategy that's used like a crew by being to split the data.  So you have this C.V. group.  And column.  So what actually what we do in the, in our production work, we did that.  So all the tiles that I presented in the morning.  So they, they are like one.  And then we have the C.V. group.  So you have this C.V. group.  And then we have the tiles that I presented in the morning. So they, they are like one 30 kilometers by 30.  So we use it like the tile ID as the group column.  So to do like this special, this the cross validation, it will be like sort of spatial presentation that will use this ID.  So you can avoid, for example, train a model with samples from one tile and validate with samples of the same time.  So, and, but it's, it's just like, you need to change the CV to group and fold.  So it's, it's a, it's implemented, it is implemented in the site kit learn.  What we did was just like a integrated integrated to this functionality.  So you can use this one, right? Yes. So you can use this one. So now I use here somewhere. Let's see.  You just use the, yeah, because here we are just using, for example, the, it's just one pile.  So we could split this point in like soup tiles and create this blocking structure.  But for, for this example, probably it will work, but we need to really define this in the data frame.  So that's a, that was a good question.  Yeah, good question. Thanks.  Let's see.  Yeah, so in here, I think you can see, yeah, now we select on the tile ID, but you can then use it to do a spatial cross validation.  And the next thing is that so all these, all these things you can use them within the package, right? So you can, now we use a K fold, but now we spoke about this group K fold, but you can apply different kinds of strategies here.  Does that make sense? Yeah, great.  Okay.  The models.  Let's see.  I think.  Where are they?  Let me look further.  This one.  Yeah, so you have, because this, this, this libraries is, they create like this base class, this base estimator. So all the algorithms that are implemented here, they have kind of the same methods.  So our package actually use all these methods training predict and things like that.  So for example, you have like the SVM so if you replace the rental forest for SVM, you will have the same structure that we are showing here, but with a different model, the different learner.  I think on top and bottom, you can see that there.  So you can use like a linear regression. So we will use more learners, but it's fully compatible what we have here in the.  Yeah. And I think you will show an example where we use multiple learners to do a ensemble machine learning.  Okay, so we have now looked at this example so our first try we didn't do any hyper parameter optimization we just train a very simple model on our training data.  And then now we will try and do some hyper parameter optimization and see if our results are actually better.  Yeah, we already saw this one so this allows is also as a learn functionality that we will also use.  Then the rest, this is kind of similar to what we did before.  But here we define our grid search so we will run over the different parameters that we want to test.  We want to test maximum depth and the maximum features and the number of estimators. So we want to test all these combinations of parameters and the rest and we use F one score that's also a standard scoring in in the S K learn package.  But here we can also define our own. I think we will do it a bit later.  We just input the estimator. And then the only thing we need to do here is just add the high parameter parameter and that will allow us to actually do this.  And then we will create search. Let's see if we can do this. So we get the similar outputs. And then now we will train.  We will actually train the model 90 times and see which one is best.  So that might take a little bit of time. Let's see.  What the computer is working on it. Some nice green bars.  So it's still running. So now it's almost done.  I think it's done. Yeah. So now we can see if the prediction actually better than before.  So we have.  What is it about a 6% increase, I think, in accuracy.  So it's, it's, it's something.  It's an increase. So that's good. Then we will do the same. We will, we can plot the.  Yeah, the, the methods to do some invitation of our results.  And we can look again at our results. We can save the model again and load it again.  And I think, did we get the output file.  Or not.  No, okay, we skip the prediction.  Okay, but we can do a prediction model. Well, let's have a look first at the probabilities and then I will, we can have a look at also these files.  Now I'm using log loss score. So this is an example of a custom loss function that we make. We also use the escalator and implementation of this, adapted to our, our case.  And we do the same thing here, but  here we also tell our class that we want probabilities.  So because we use this random force with a lot of different predictors we can see if they are corresponding to each other or not and if they are all pretty sure about the class then the probability is high and otherwise, the probability is low,  or could be lower. So we make prediction again.  And let's see if there's different results.  We need to train it of course.  So now we use a different loss function so we will.  It's a bit different to compare it.  So here we have now our accuracy metrics but then get towards the last metric that we defined for each class again.  So that's, so we get from the model we get different types of accuracy metrics that we can all access through through these probability metrics variable.  And then we can also print them.  All right.  And then we can see if this loss function actually gives us a better results.  By looking at the, by looking at the prediction results.  So now I'm applying the same function so to predict with the layers.  But here I'm putting all the years. No.  So now we're still looking at just 2018.  So here's the same function to find the right data and to put all that together. And then we're loading it into the landmapper class and producing a prediction but now not just the class but also the probabilities for the different classes.  And we also provide the plot roster function to actually have a look at the results.  So, yeah, it's of course, very rough interpretation but if you look at this one compared to the other one.  Let's see where it is.  You see it's a bit cleaner.  It's not sure if that's better but at least you can visually appreciate the difference in the result.  So if it's better that's of course matter of further interpretation and looking at the data.  But now we have a bit more more at hand to actually do these interpretations.  So we can now also have a look at the, at the probabilities.  So I think these are related with.  So one is related with zero in this case. Yeah, okay.  So I'm not trying to visualize the probabilities for the Morrison heat length that we just produced using the landmapper class.  So, here we see the probabilities for this area.  So again this is where we are now.  Let's see.  So this is the bargaining area this is bargaining town and here is the conference center.  So now I can have a look where it's likely to have more in heat length.  Yeah, let's look what's this.  So this actually airports.  So at least now we can see what what what areas have impact on our, on our predictions.  And I should actually what I like to do is, because it's still the easiest way to to act to have compared these things to go to our viewer and see, you can also see the Lancet data over time.  So that actually gives us a lot of information on what it is we actually see in our training data directly and then you can. Yeah, it's not.  You can probably also load it in GIS but this one is like really, I think really user friendly to go to different years.  We have to.  Okay.  This one. Yeah, yeah.  That's it.  So here we can have a look again.  And now we have layer coniferous forest.  So let's have a look.  If this is corresponding to what we found.  So here we are.  It's a tricky class, because it's very much like I think many other classes.  But hopefully our model also picks up these other classes and actually produce higher probabilities for these so that even though we see high probability here, maybe other classes still have higher probabilities there.  So let's have a look. We now created our own prediction here.  Let's see.  Let's stay close to where we know where we are.  So this patch for instance.  Yeah, that's different.  So now I'm observing that our predictor kind of finds a patch of heat land here or at least a higher probability than that we have found in the prediction codes that we used for the  Yeah, that we used for this. So that's interesting I think you know this is. Yeah, just some interpretation of the results that you can do and I think all these tools they really help you with.  Yeah, visually interpreting also your results and see what what you find in your training data.  So that's very interesting.  Let's see where we're, and you can do this for all these classes so this class five for instance that's going to first forest. So this makes a lot of sense if I look at this because if you are here, Wageningen then this is the veil with that's the biggest force in the Netherlands.  So for me, I think okay this is at least there is quite some some truth in this prediction because I know this is forced area. Of course, there are many much more research to be done to confirm this.  And then I think the under all you want to show the people the ensemble machine learning. Yeah.  Yeah, okay.  Thank you.  So I'm not sure any questions about this part.  Please.  No. Okay.  Yeah, so.  And now we have like the, the probability results.  And I would like just to.  So, please show how to execute it.  And I would like to explain this report here.  And the metrics that we generated so.  Here you can see that this is, you can see this report because Chris instantiated the landmapper with the verbals through so even for example for for us that we are developing this this API and we are using like on a daily base.  It's really important to follow the process so follow how what it part of the, the, how which part of the workflow it's executing.  So these verbals it will like explain everything that happens in the landmapper.  So, the main, the main points here it's really, you can use all the class that are available in the signkit learn. So, random forest for learners but you can also use SVM you can use a Keras classifier that it's a implementation compatible with signkit learn.  So I will use it, but you also can use it individually. And you can use different types of validation strategies.  So here we are just using mainly like cross validation so it's, it's, there are a lot of discussion if the cross validation it's, it's the right way to evaluate it. But what we decided when we were implementing the library is inside of the library will mainly use cross validation.  So if you want to validate your result you can do it outside so it's just a matter of have independent points and do the predictions using like the predict points method.  So, Chris presented the, the interface here so you can see that you have these predict points. So if you have like independent points you can manage it outside.  So, and, for example, you here we are using these Lucas points, combined with Korean Korean land cover, and we are creating our data set. And for the production work actually we use it almost 6 million of points so it's much more points that we are presenting here in this training  because it here it's just a matter of how we demonstrate these functionalities. But, for example, we have the model train it now our production model like to produce the maps that are available now.  And we can conduct, for example, I independent validation strategy and we can spread points all over the Europe and expected it by visual interpretation or even go to the field if we have money and the context of the project, collect the actual class of it and just use these  points to do another prediction so it's a completely independent approach that we, it's not really related with the training and how we are implementing these inside of the model inside of the land map.  So, you have this, this possibility. But what it's important here is, we are using the cross validation, because it's a kind of, it's a kind of metric that allows you to evaluate your training process, how you are capturing the variability  in the class how your features are really like correlated with the, the expected variable and you can use this, like a reliable way to see if the model it's doing a good job, but to provide like a real validation of your result.  And you have like independent data set and it's better provide like a proper sampling design so spread points considering like all the assumptions of the statistics and like a random sign or stratified design to do it but of course, sometimes you are limited, you  don't have like budget enough to collect points to produce new data. So you need to work just for example with cross validation but in the literature now you have this kind of discussion and it's, and I personally I don't believe that a cross  validation it's able to provide like a real and a proper validation for your prediction so it's a nice way to understand if your model it's doing a good job. It's a nice way if you, to understand if you are optimizing it, but to provide the real validation it's  like independent data set so it's, it's, it's what we presented in the beginning of the presentation so you actually you need to have no new data so and, and you can try to split it, but for for the context of the land cover it's it's, it's how it's the state of the art, sampling  design to provide a better validation and a proper validation for your product.  So, you, you have it here, not all these aspects but you have the possibility to use what we train it now to predict new points, not just  the, the, the, the, raster raster files. But if you back here. So you can see that considering this cross validation strategy that we are using here it's a five fold  cross validation, the landmapper, it's actually  dual optimization approach to find what are the best  hyper parameters.  And it's find the best, it's something like vague.  So basically what it's doing,  it's calculating some currents metric.  And here you can see that Chris explained it.  So you have here it's a F1 score weighted  and it needs to be weighted because you have,  it's a multi-class classification.  But for example, if you click here,  you can see that all those metrics are really available  in the signkit learn.  So we could optimize also by the currency  or by the precision and by the recall  that are actually in the context of remote sensing  are like user and producer currency.  So it's possible do it.  But so it's really like it's a modular approach.  So you have like validation strategy defined,  you have which parameters that you want to test.  So all the parameters that I put here,  it will be a combination of all these parameters.  And you have different strategies also.  So I always prefer to find the parameters  because it's sometimes you really need to think about  what not all hyper parameters needs to be optimized.  And so, and it depends of each algorithm.  So define these values actually force you  to understand a bit more about these learners,  about these models and select actually what are like  the best, not the best, but the proper parameters  that you need to optimize.  So in here, we are optimizing the number of trees.  So it's something that it's a common standard  in the most part of machine learn applied  to land cover classification and other domains.  But if you provide like, I don't know, 60, 80 trees,  so you should have a good result.  So considering like dominant class, hard class.  So it's a kind of standard value.  But of course, if you put more trees,  you will provide, you will have a higher computation time  because each tree needs to be computed,  need to be trained, need to be predicted.  So, and this number, it's actually,  it's really high correlated with the performance of the model.  So for example, if you have 40 trees  and you feel you have 40 course,  that's not our case here, definitely, we just have four.  But if you have like 40 course,  each of these trees will run in parallel.  So the process will be way faster.  So it's this hyperparameter optimization,  you want to optimize to find the best model,  but there is a plateau.  So you need to be aware that sometimes  you are just adding computational time  and it will increase, I don't know, 0.5% in your records.  But for some applications, maybe this 0.5,  it's really important.  So if we are doing like with, for example,  medical applications, it's life.  So you need to optimize at most,  but it depends off the way and the domain  and the application that you are trying to do.  So you have this, for example,  flexibility here and even for land cover  and for geo-spatial classification,  maybe you need to really improve the accuracy at most  and process more with a high computational cost.  So it really depends.  And here you have like these other three parameters  that are actually the, what is the maximum depth  of the each tree?  So if you think that these three will like be trained  like you will split the data set  on each node of these three  and this maximum depth will be actually,  the default parameter here, it's non.  So it will grows till like separate all the samples  in the training set.  But sometimes you, if you reduce this depth,  you will have like a shorter trees  and maybe it could help for the probabilities  because you will have like more fuzzy output.  And if you combine all these trees, maybe it will help.  But it depends off your model  and the way that you are training.  Not the way that you are training,  but your training data set.  And the maximum feature, it's actually  how much features it will be analyzed for each split.  So for example, if you put here 0.5, we have 86 features.  So it will use half of it to define the best variable  to separate the training samples.  So, and for example, sometimes I use it like one here.  It's totally random.  So the random forest will be trained like for each tree  and each node, each part of these three,  it will select randomly one variable  and it will try like split.  And maybe this variable, it's meaningless.  So maybe this variable, it's not really performing well  to separate your point samples,  but it will create this node past to another level  and select other variables.  So maybe you could have like a really deeper trees,  but it could help.  So it's difficult to predict it.  So you need to really think in some parameters,  think about what are your computational capacity  and think in how you will do the predictions.  Because I don't know, maybe we can have a random forest  with 1,000 trees, but how we will run it.  We have a server to do it  or maybe it could produce a very long computation time.  So you need to balance all this to define these grids or CV.  So it's why I really like to not use them,  there is one strategy that you just put some random value  between like one range.  But I think it's nice to, if you think about it  and really try to put some values that are specific  for your domain, for your problem and for your application.  So mainly here, what we are seeing is all the  so you can see that there are 18 candidates.  So it's mainly three multiplied by three.  And so yes, three, three whites should be less.  So 18, yeah, three.  Yeah, three by three, nine.  Yeah, multiplied by two, 18.  So actually, if you think in all these combinations,  you have 18 combinations of hyperparameters.  So it grows fast.  So I don't know if I put like more two parameters here  in each of these options,  it will be more than 100 combinations  and you can quickly blows the process.  Not here in our case because it's a small dataset,  but mainly for all these 18 candidates,  we are using a five-fold cross validation strategy.  And so actually we are training 90 models.  So, and here you can see the result of the F1 score,  weighted F1 score for each of these combinations.  And here we are optimizing it by the higher value.  So, and the best parameter is this one.  So maybe we could optimize a bit more,  but for this dataset and for the random forest,  we can put more trees if that was the,  maybe that could help, but maybe not too much.  So, yeah, I think this,  yeah, I think here we stop for a break.  And I will back for the second part  and we can analyze better the probabilities  and predict other years.  So, and there is, I don't know if there is questions,  there are questions.  Nothing to chat.  So, okay.  So let's have a break and we back in half an hour, right?
B1_0_1571.png B1_1572_1691.png B1_1692_2135.png B1_2136_2195.png B1_2196_2243.png B1_2244_2543.png B1_2544_2939.png B1_2940_2987.png B1_2988_3143.png B1_3144_3311.png B1_3312_3491.png B1_3492_3539.png B1_3540_3587.png B1_3588_3815.png B1_3816_4115.png B1_4116_4307.png B1_4308_4667.png B1_4668_4739.png B1_4740_4871.png B1_4872_5711.png B1_5712_6167.png B1_6168_6515.png B1_6516_6611.png B1_6612_6935.png B1_6936_7031.png B1_7032_7391.png B1_7392_7691.png B1_7692_7859.png B1_7860_45083.png B1_45084_49787.png B1_49788_52667.png B1_52668_53495.png B1_53496_53567.png B1_53568_53663.png B1_53664_53723.png B1_53724_55811.png B1_55812_55895.png B1_55896_56171.png B1_56172_56531.png B1_56532_57047.png B1_57048_57131.png B1_57132_58283.png B1_58284_58631.png B1_58632_59255.png B1_59256_59423.png B1_59424_60767.png B1_60768_63023.png B1_63024_70439.png B1_70440_71027.png B1_71028_72107.png B1_72108_75287.png B1_75288_76223.png B1_76224_78827.png B1_78828_79223.png B1_79224_80435.png B1_80436_80579.png B1_80580_80723.png B1_80724_81215.png B1_81216_81347.png B1_81348_81527.png B1_81528_81791.png B1_81792_82511.png B1_82512_83375.png B1_83376_85199.png B1_85200_86423.png B1_86424_87611.png B1_87612_88019.png B1_88020_88079.png B1_88080_88223.png B1_88224_88547.png B1_88548_88595.png B1_88596_88667.png B1_88668_88859.png B1_88860_88955.png B1_88956_89003.png B1_89004_89291.png B1_89292_89819.png B1_89820_89927.png B1_89928_91247.png B1_91248_92051.png B1_92052_92543.png B1_92544_92831.png B1_92832_93179.png B1_93180_93347.png B1_93348_93419.png B1_93420_93659.png B1_93660_95375.png B1_95376_95591.png B1_95592_95675.png B1_95676_95795.png B1_95796_95915.png B1_95916_96287.png B1_96288_97043.png B1_97044_97235.png B1_97236_97343.png B1_97344_97487.png B1_97488_97547.png B1_97548_97871.png B1_97872_98375.png B1_98376_99191.png B1_99192_99503.png B1_99504_99587.png B1_99588_99779.png B1_99780_99899.png B1_99900_100091.png B1_100092_100355.png B1_100356_100595.png B1_100596_101087.png B1_101088_101147.png B1_101148_101927.png B1_101928_102071.png B1_102072_103259.png B1_103260_105143.png B1_105144_105515.png B1_105516_105743.png B1_105744_106127.png B1_106128_106883.png B1_106884_107867.png B1_107868_107915.png B1_107916_108011.png B1_108012_108071.png B1_108072_108887.png B1_108888_109535.png B1_109536_111227.png B1_111228_117095.png B1_117096_117215.png B1_117216_118031.png B1_118032_118211.png B1_118212_119099.png B1_119100_129611.png B1_129612_131099.png B1_131100_131579.png B1_131580_133085.png
Let's back to the training sessions. So this in this afternoon, we will use two slots to present about the spatial temporal machine learn in Python and how we implemented it in the context of the open data science Europe. So the presentation will be, we will start with me and later Chris will join to do some code demonstrations as we are doing for all the sessions. And we'll have a break and after that we will back and we will discuss more some about the ensemble machine learning. So and the, to the approach that we implemented now in the EU map. So before to go to the code, I put some slides. So I prepared these slides to provide some conceptual background about machine learn. So how we it's important to provide like a common understanding of what we are doing in a general way, because of course machine learning can be used by different applications. It's kind of buzzword now. So everybody's talking about it. So considering this general introduction, I will explain how to do it specifically in using geospatial data. So what are the, what is the approach? What are the data? How we can implement a processing workflow to do it? And specifically on that, I will show two demos, two real use applications that classify a land cover. One, it's purely spatial. So that I implemented in my PhD. And now it's spatial temporal and I implemented, we are implementing it in the open data science Europe. So I see it as a kind of evolution between this purely spatial to spatial temporal. So there are some important aspects I will explain about it. And after this introduction, we will have a hands on like a demonstration and we will actually we will execute the code using mainly the EU map package. So there is a class that we've created, landmapper, where you can implement all these machine learning processing specifically for geospatial data. So let's start. So I like to, I would like to start with the definition. So what is machine learning, and if you think in a nutshell, it's essentially a subfield of artificial intelligence. So it's not the same thing. You have more subfields inside of artificial intelligence, like robotics and fuzz logic. So machine learning, it's a subfield inside of it. But most important, it use techniques to learn from the data and make accurate, accurate predictions. So the whole point to use machine learning is train over like some specific data and do predictions in new data. In the field, we call it like a generalization. So you receive new data that the model didn't see and you produce good predictions. And there are several ways to implement it, different algorithms, but in essence, it is. And other important part is without being explicit programed. So if you think what it's happening inside of machine learning model, you can code like hard code, all these rules. So if you think like a developer, I could get an image and do a lot of rules myself to try to identify, for example, where is water and work with the data, try to find some values. And I can code that. But if you think in the way that machine learn do it, it implements it for you. So it will create all these parameters, look in the data and try analyzing the data and try to establish it like automatically. So without being explicit program it. So if you think in the artificial intelligence field, you have other subfields, so machine learning is one of these subfields. And inside of machine learning, you have like other buzzword today, like deep learning. So here we are, we will not talk about deep learning. So we need to, we could have other session about it, but we will use TensorFlow. So it's not every time that you use TensorFlow that you implement deep learning. But and today all these concepts are kind of mixed. So for example, in artificial intelligence, you have like a natural processing language subfield. And today you have pretty good algorithms and really accurate algorithms in deep learn to deal with that. So some of these subfields are really changing too much. And deep learning, it's a really hot field in the research. And there are so many algorithms and really nice implementations of it. But mainly you, when we are talking about deep learning, we are talking like processing graphical card interface, processing unit like GPU, convolution neural networks and all, it's more related with it. We have like a recurrent neural networks also. But here at the end of the training session, we will use a regular artificial neural network and it will run on forest and with the gradate and boosted trees. So in this session, we are actually inside of the, just of the machine learning, not actual deep learning. So now if you think in the machine learning, you have mainly three like categories about algorithms. So we have the supervised, the unsupervised and the reinforcement. So the supervised, it's mainly when you provide like training data and using this training data, you already provide like the answer. So you need to have like raw data can be image can be text can be a table. It doesn't matter. But you need to label this data. So what it means, it means that you will look this data. So some human being will look this data and it will create like what is the expected result for this data. So if you think like in an image, we need to provide for the satellite images, where are the class that we want to map or the values that we want to map. So for example, you could provide like coordinates points, specifically on top of the image showing that this is a patchery, this is a conifer forest and things like that. Or you could provide polygons also. But you need to provide it. So you need to have a data with label, so label data to do the training and the machine learn will try to match and to analyze all this data to provide like expected results and you can compare with that. So here mainly we will use it. You have the unsupervised learner. It's more like machine learning. So you don't need like training data and you can work with the raw data and try to find some like hidden structure on it. So for example, for the spatial classification, you could try analyze purely the values of that image and try to aggregate values that are similar. So if you think maybe a pixel in a forest, it's in specific part of the image, it's close and or it's related has values related with pixels on other part that are forest also. So it's possible. But sometimes the most part of the times actually they are not really accurate. So the accuracy, it's better if you have the label data. But it's there, there are researching, there is a lot of research in this field also and you have the reinforcement learner. So machine learn this reinforce you learn by mistake. So you have like actually an agent that try to do something and you have some way to measure what to analyze if that action produce or not the expected result. And it will learn considering the response of different attempts. So I'm not familiar with this approach and you again, you can combine it so you can have like a supervised learner and you can after that you can have like a reinforcement learn. So but it's important to be aware of these these fields and these concepts. And here, mainly we will work with the supervised machine learn a specific classification algorithm. So we are not doing regression. And if you think in a classification, the difference between classification regression mainly we are producing like factor values or like integer values or class values. And in the regression you are the model will predict like a continuum value. So it's more like to predict some variable. So in our case, the land cover, it's actually like a class, we have the same exactly the same value. It's an integer, but we have the same value for different pixels of the images. But for example, if you think if we we could train a model to predict the precipitation, for example, and to predict precipitation, the output will be like a continuous value. Each pixel could have a some level of the precipitation and the this is more like a continual value. And if you need to use a regression to implement it. So and the other part it's it's about it the program. So inside of the the model, you will have like what we call like the parameters. So the parameters are actually the how the model will implement all these rules, all the training rules to split and to analyze the data and predict the expected result. So to predict if that pixel, it's a class, I'm talking about image, but for example, you can do the same thing for text. So now in nowadays, for example, the Google translate do it. So it analyzed some of the whole structure of the text and predict the output as a new in another language. And it's mainly it's the same concept. So you have the input data set, it's actually text, and you are predicting and the Google translate predicts it as a as a text, but in other language. So and if you think in the way that it was implemented, any anybody created like some anyone created some hide and rules inside of it, the model learned it considering millions of training data in the case of the Google translate, but you can have it for all the for different kinds of applications. And one important concept is when we are talking about the model, the parameters are like. We train the parameters. So if you think in the random forest, it's basically a decision tree. So you will analyze the data and this data could have like eight, 10 features, 10 variables. In our case, it's 10 images. We will use 84, but it could be like any number of covariates and features and analyze this picture, this this features and this pixels. The model will train like a tree and each part of these three, it's a decision tree, it will like try to define to divide this data to classify in pasturing in conifers forest. So you the way that this decision tree is created, this is the like specifically the rules. These are the parameters. The hyper parameters are mainly the they are related with how you control the learning process. So for example, if we use 100 trees or 10 trees for for the random forest, this is the hyper parameter. So number of trees is a hyper parameter inside of each of these trees. We have parameters that will analyze the data and it will split it and it will generate the output. So this is the the parameters of each learner and different algorithms, they have different parameters. But mainly what you need to remember is the parameter, it's actually they are learned from the data. So and it's it's it's automatically derived considering it's a data driven process. So you don't need code anything inside of it. You just need to code outside to control the process. So this is a expected machine learn supervised machine learn approach. So we have some raw data using this raw data, you can have features. Again, this could be anything tables, text, images doesn't matter. And you have labels labels are like, what are they? What is the expected result? Considering this raw data. And of course, the raw data, you can derive just some features, you can have like a bunch of raw data, but just part of it you are using to do machine learning. And you have like the model training and the model evaluation. And with a training model. So after the training, you have like a model, a training model, you can save it, you can load it. But mainly you can use this training model to do prediction over new data. So that all that effort creating the labels and create the expected result. And there it needs to be currently actually it needs to be done by someone. So human human being and me. And you can use it to predict and to generate predictions. So this is the expected workflow, but for the real world applications, we have more something like that. So it's not so beautiful, but you have the when we are talking about real applications, you have like a way more complex, for example, data and raw data than this benchmark data sets. And when I'm talking about benchmark data sets, I'm saying like, if you download, for example, the TensorFlow, you have like some digit numbers, like images with digit numbers, and they are really organized and pretty and ready to go. So you just need to get this data and do some predictions and you already have the labels. But the most part of real applications, they don't have the data really organized like that. You need to prepare the input data set, you need to prepare the labels, you need to expect to prepare the expected result. And you need to work all with all the modeling part. But at the end, you need to do predictions. So otherwise, you are not interesting, just create a model and use this model to just to predicting just a portion of the data. What is the end goal considering a real use application here? And for example, in our case, the end goal, it's really mapping the length over, map the length over classification and the continent of Europe. So we did all this effort, but at the end, we need to predict it. So we need to get this, this model and really scale it up and predict all the pixels of Europe. So there is a way more complexity than just like a simple workflow. But mainly, if I could like put the two most important aspects, you need to prepare the data before to really start doing modeling, starting to machine learn. And you need to be aware what you plan to do with the final model. So what will be your prediction strategy? So you can put it in a, I don't know, in a service and the users can access the servers as for example, we have for the Google Translate, it's a service for a machine learning model. And everybody can go there and just put some text and it will translate. So considering a real use application, you need to be aware of how you will serve your predictions. So you can do all the predictions in-house as we are doing, generate the maps and put it available to the people who use the maps or we can really serve the model and to the other users really access and do new predictions. So there are several possibilities on it and it's important to be aware of these possibilities. So but let's back to our simple example and we can expand it for the geo-spatial data. So now if we specifically explain it in the context of the geo-spatial data, this is more or less the pretty way I say it. So you have like the raw earth observation data. So this earth observation data as I was explaining the morning, so you can have like the Landsat data, Sentinel data, there are a lot of satellite images available, but you need to download it, you need to organize, you need to remove the clouds, you need to do several pre-processing steps and to produce analysis ready earth observation data. So and using this analysis ready earth observation data, you actually need to provide now the labels. So what you want to classify in this image. So you want to classify like soil organic carbon, you want to classify land cover, you want to classify, I don't know, just one class, pastry maybe, or you want to classify water. So you need to provide the labels because the performance of the most, the better models that are available now, they work with supervised classification. So you need to provide the samples, the training samples. And the training samples, it can be like points, lines, polygons, it doesn't matter. A sample, it's just like a vector that you will overlay with the data, get the pixels and put inside of the machine learning. So here in the training session, we will show the like an overlay of data, but in the U-Map, so look at, did talk about it in U-Map, you have like the functions to, I don't know, get one point that samples that you collected in field, for example, and overlay these points over like raster data, different raster layers. So we are, we, in these training sessions, we already did it, so we will work just with the overlay of data, point data, that for each point, we extracted all the pixel values. So and considering it, we will train models, so we need to train a model, we need to evaluate this model, so there are some strategies to do it. And after that, we will have like a trained model, and when I say like trained model, it's a ready to go model, so it's a model, it's a file, it's a model that it will be able to do predictions without do the training again. So we will show it for you in the code. So mainly when you do the training, you have like a model, and this model, he has like, he knows how to process new data and predict like the land cover. So and he did learn considering just the samples, but you can use to do predictions in different parts of your data, of course. So there are some limitations with that, but then go with, is you really, for example, train a model to do the prediction in the whole year. So as we are doing here, and we will, and you can load this model and you can skip all this training process and just predict. Yeah. Yeah. If it's straight going, that's the idea that you will be able to process analysis, right? Rather than the route. No, that's a good question. Yeah. So actually, you need this rule, this arrow here, you need to read from the analysis ready data. So because, yeah, I will fix it in these slides. Thank you for noticing it because this, the training, it's really sensitive. So if you, if you change a bit the pixels, you will change completely the predictions. So it's, it's, there are some techniques to really try to put, like you can simulate noise in the analysis data and to try to predict raw data, but it's, it's more, it's a different type of research. And here we are predicting analysis ready data. So this is one of one example. So I did it in the, my PhD. So for, I worked in Brazil using the Landsat data. So mainly I, we had like more than 30,000 points for each year. And these points were visually interpreted by three specialists and they classified as pastry, forest. So and they are, you can see the points here. They are really spread all over the country. So this is a, we, and in my, in my PhD, I was part of the GIS and remote sense laboratory in Brazil. And actually we generated this data from scratch. So here we actually develop all this. We develop a sampling design. We spread the points all over the country and we develop a solution to help the specialists to check the pixel values and see if it's pastry, if it's crop and things like that. So in this context, it's, it's, it's not like it's really specific, but the, what it's interesting here, we have samples for the whole country and for the, all the years. And that's, it's not the main part of the applications. So, but even have, so considering that we have points for all the years and for all the country, you can see here, this little square, little squares here are like, those are the Landsat scenes. So we have 308 tiles, Landsat scenes to cover the entire country. And we have 34 years. And what we did here were really, we, we, we, we trained one model for each scene. So if you think in the machine learned way, the training process occurred by each Landsat scene and by each year. So at the end, we trained more than 2000, almost 13,000 random forest models. And it, and we mapped individually all the years. So, but if you think we have all this data, so it would be possible training one single model. And what I mean with that, I mean, I can get all the points from all the years and for the whole country and put all these points in a single machine learning model and train it. So it will be millions of points. And but maybe it could produce a better result. And but they're mainly there are two problems with that. We need to have the capacity, software and hard to do it. So it's not easy training a machine learning with millions of points. And we need to have our monizer data. And if you remember, my first in the morning, I explained that if you work with just with the raw data with the lens, we have three sensors. These sensors are different. So one image of 285 will not have the same pixel values, the same reflectance values from like an image of 2020, accurate by Landsat eight. So you need to work with harmonize it, the Earth observation data. So and they are true products at least probably there are more. But the main products to do it, it's the glad Landsat R&D. I talked about it during the morning and there is this other product that combines landsat and sent in out shoe. So in produce like one harmonized data cube, but it's just with Landsat eight. So we will have just since 2013, but I think they process it just from 2015. So this is mainly this is mandatory to try to create one single model for all this to classify for example, multiple years and different parts of Europe. So in the open data science Europe, we are doing it mainly. So we took all the Lucas points and that it's a basically a well-known field survey here in Europe. And we combine it with some generated points from the old from from the other land cover maps Korean land cover classification. And mainly we are using all these points to train one single model, but it's important to emphasize that we have the capacity to do it. And we also have like a harmonized earth observation data as the glad Landsat R&D. And without gaps and all these parts that I was explained during the morning. So at the end, we are predicting dominant class probabilities and uncertainty. So in the end of this training session parts to you will produce it also in the Python and notebook. But we are also using more than one learner. So random forest, it's already a sample learners because inside of a random forest, we have several trees. So decision trees and it's why they call it like a forest. It's multiple trees. But we are also using the gradient boosted trees and the artificial neural network. So we will show how to do it in Python for the tiles that we prepared. And yeah, now it's time to go to the code and start to really do all classifying the land cover, considering the data that we prepared. So just to overview of what we will do mainly, we will train a regular random forest, pretty straightforward without any type of hyperparameter optimization. It will be just a purely random forest and we will predict it to see how it works and to understand how we map it's doing it internally. Later the second example will be the random forest with hyperparameter optimization. So you can test different combinations of hyperparameters to really predict it and benchmark all these different parameters and find what is the best one considering some evaluation metric. And the second part, we will train the random forest with probabilities. So if you think in the random forest, you have multiple trees. So each tree will produce like an output and you can use this output to derive like a kind of a probability. So and this probability, it will, it changes completely the output of the algorithm. So we will see it, but instead to just have a hard class, a dominant class, so it's pastury, forest, you can have like multiple probabilities and these probabilities will show, for example, maybe there are some, there is an area where you have like a not a high probability, you can have a high probability just for one class. So you have sure that the model is pretty sure that this is, for example, coniferal forest, but you can have like other pixel where the coniferal forest is 60% and the broadleaf forest is 40%, 30%. So there is some confusion there. And if you think in the image, man, it kind of makes sense because you had some limitations with the data that we are working. It's not possible really get one pixel and have around 100% sure that inside of that pixel we have just one single length over class because we can have in a transition zone. So there are several complications of it. And at the end, we will expand this and put more learners, so more three learners and explain how we are deriving the uncertainty and how we can improve this result. And this is actually our current workflow. The code that we will show here is actually the same code that we are running in our servers to produce the length over map available in the open data science Europe. So now I will pass the floor to Chris. I don't know if there is any question for this introduction. No. Okay. Should I take the microphone? Yes, please. Hi. Thank you. All right, let's look. Okay, we've got the virtual box to work. Oh, am I unmuted? Can you hear me? Oh, good. We're all set. Okay, thanks Leandro. So I will show you a little bit of the code and we'll see if you can also replicate some of this. Who is able to also join me with the code in the machine? In the virtual machine? One, two. Two people. Okay, so for the rest I will just show the code and then in your own time you can have a look if it's also applicable to your projects. So this is the Jupyter Notebook that we will look at today. And then here we see the example of the interface, the online interface that we use to also publicly make it publicly available. And here we have the reference to, because much of this work is from Martijn's work that's also here, which will soon be publishing on this as well. So that's also interesting, I think. So first of all, we load the main packages that we need. I will reset the kernel so you can see how it's working. It should be working. Let me also clear the outputs. Is there anyone here who worked with SK Learn libraries before? No one. I think I know some people who did. That's okay. Okay, so you'll be recognizing some of this code here. So we will be loading the EU Map package, which is also, as Salerno mentioned, the main package that we, the current workflow that we use for the data that we provide. And we use some standard libraries, something to find our files in the system, and to load these files into the environment. So let's load these packages. Then I will define the tile that we use, so we provide two example tiles that you can try. And this, I just run this to be able to find the data on my drive. So here we have the file system, and I put the directory of the data. So that's this directory. Okay, so let's run that. Then I have some training data that is here. It's a geo package. So that's basically a lot of these points that are mapped that are we know, okay, this is this land cover type that is all put into this database. And then we will load it using the geo pandas package. So now this data is loaded into the variable called points. So here we see just a regular table with Yeah, all the information. So here we see for this, for this point, we know the class is 112. And then this is the value that the distance to the coast. This is the weighted distance to the coast, I think that Martin developed. But also if we look further into the data, then we see the land set values, and also some the names and the survey year. And because this is a year, this is a, yeah, this is your reference data. So we here see geometry. And the nice thing for the geo pandas package is that we can also see what what what reference system is used. So this is just the, this is a geo pandas data, data frame that has this, these functions to to find all this information. And actually see what my reference system is for these points. Because if I if I look at these values, it's not is not the regular coordinate so here you can see how how should I translate this if I want to plot it in another kind of software. And I want to get some information about my data. So for instance, so I just make a kind of a bar bar bar graph about some of my values. So here for instance I have the class name and the survey year, and then I want to see how many points do I have in my training data just to get an idea of what, what I have here. So I have a lot of coniferous forest points and more San Heathland and agriculture with natural vegetation. And here I see when these points were measured so I have most data in 2018. And this we can, you know, I generally I would do a more in depth look at these data but for the, for this presentation I will just go ahead with looking at the random forest. And then I will, I will focus on doing just the spatial and then the under a will later also include the spatial temporal so then we will see. I will be focusing all the years. Okay, so I will look at the spatial temporal data actually. So here I am also including some, some escalern functions. So this is actually to do. I think we use. Yeah. So this is for a bit later where we use the hyper parameter optimization, I'll come back to that. Then the random forest layer classifier that's one of the classifiers that's included in the escalern package. And there's many, many types of classification algorithms, and they're all kind of standardized so we try to also enable you to use all these different classifiers which makes it really versatile in how you can use it also on your spatial data so I think that's that's a very cool feature of this package. So the K for this to do, yeah, K, K amount of faults on your data to get a, an interesting evaluation metric. So here I will initiate the estimator the random forest estimator with five estimators. I'm going to select points just from this, the tile that has my tile ID that I have defined before, and I will use a five fold cross validation for the training. And then I define all the, all the variables that I want to use for my training. So in, in my data set, all the columns that we saw, I will put the prefixes for all these columns here. And then I will set up the, that I know which, which variables I'm using to train my classifier. So this is something that, that was written into this package so you can really be quite sure about what you train on and how you, and it's easily configurable. So I have my target column, that's our land use class. So I will load the land mapper class with all these configurations. So I have my points data for my data frame with all the training data. I have the variables that I want to use so it will be selected from this data. And then I have the target. So what should be trained on and then the estimator, the, yeah, the class that will handle the faults and another parameter where I just select classes that have enough training points. So I'm just going to look quickly at this one so it will select only the classes that have the highest amounts of, of training samples because for these ones we cannot reasonably believe. I think that we can make a good prediction because we don't have a lot of training points. Can I just explain it because we included this parameter because when you do a cross validation, you need. You want to have a look. Okay, so when you, when you use like this, but this parameter, it's important because when you do a cross validation. Mainly you will divide like imagine you have all these points you divide in five regular groups, but you need to have enough validation points in each of these groups so I don't know maybe you have just one or two points for some class. And this point will be present only in one of these folds in one of these regular groups and sometimes some algorithms that doesn't manage well with that. So, you have like, you need to provide it because it depends off the, the, the algorithm you can classify with less amount of samples. But for example, I know that SVM has problems with that. XGBoost also has problems with that. So there, there is a limitation with the way that the validation, the cross validation it's implemented considering this regular fault because you need to have all the possible classes in each of these folds to do a proper validation. Okay. Thanks. So, again, we, so we also focus on that it's usable in a production environment. So maybe sometimes you don't want all the, all the outputs to be printed to the, to the, to the terminal, or whatever and then you can also put, you can switch off all the outputs. So that's also useful feature I think. So let me see if I run this, I run this. I run this block of code. So I get some information about what the land map is doing. So it's removing some samples because it doesn't, it doesn't comply to this condition that I specified as they under just explained. You can just explain that this is actually the percentage right. The percentage percentage. Yes. So it's the percentage for each class. Yeah, just just back to that logic just because I think it's easy to this one. Yes, just, yeah. So it's, if they are less than 5% of the total of the samples they will not be selected. Yeah. So they, these are less than 5% of the total of all the samples that we have. So these are the original classes, the numbers and then they are remapped to this because some of the packages in as killer and they need us to have, they cannot handle these kinds of numbers to do classifications. So then we will train train this one. I'm putting this number so that we have the same selection of folds for our training. So we can have a bit similar results if you run it on your own machine. Then I can look at the accuracy. So here we can see all the accuracy metrics. So we get a 56.6% overall accuracy. And then for each individual class. So if you want to know which one it is precisely you can have a look here, how they are mapped. And we will also print a confusion metric metrics that will show us some in a bit more visually appealing way. So here I am just creating because the other names are in the, are in the data that we provided. So I'm just creating a dictionary to translate the number that's mapped to the actual name so it's still understandable to us. So here we can see the true classes and then how the model is predicting them. So for instance, let's see, sports and leisure facilities are confused with broadleafed forests sometimes. Or yeah, that's happening sometimes. So this, this you can use to get some information about how your model is performing and maybe where you need to see if you can do some feature engineering to increase your accuracy for specific classes. Let's see. So then you can also access this in in a tabled way. So what you expected the class to be for every prediction that we did. So then you can generate any type of accuracy metrics that you would like to do with your own functions. So after training, then we can save this model so we can use it in a production environment or something somewhere else, where we don't have to retrain it every time. So I'm just going to save it to this, this file. So we can find it here so the, it outputs where the location is. So I'll show you where it is. So this is actually my model now so I can actually just put it on in somewhere send it to someone and he can also try to predictions with the same thing using the EU map package. And now I'm just going to reset the kernel to see if I can actually load and do predictions again with the same, same model. So I have to reload all these packages because I just killed the whole instance. And then I'm going to define the parameters again, because everything is gone. And then here I'm going to load my model again. So this into this variable. So, we prepared all the data into these directories here that Leon would just talked about so some timeless layers and also the, I won't open it because it's many files so it will take some time. But there's a lot of lens at all the lens at bands that learn would just explained for every season for every year they're all in here. And for all the quantiles, but these two functions they will find all these the past for a specific year that I want to predict so now I'm just going to predict one year. So I will find all these layers here, I just print five of them so for one year I have. Let's see. I put them together so I can import them into into my class into the landmapper. So, so I have 86 variables for this. For this prediction that I need. So that's just, I'm just, I have all the past to all the variables that I need into the in this variable. And then I need to rename them because the original ones they didn't have a year there. So now I have in every name in every data that has a year component I have the year also in there. So I want to only get the 2018 data. So, yeah, replace the year with 2018. So I can rename all these files so my, so the model can find the right data to predict on. So it loads the right files to do the prediction. So here I put all these, all these parameters into the model line. I don't, I'm not interested in the output. We just looked at it. So we're not interested in in how the renaming works, because it's a long list of names that will be replaced so it's a bit. That's a good point. So let's see what happens if we show it makes more sense actually so you see what. So here we see that the original name that I had in my model for this variable it was this one so it has no year components, but I want to find this one where it says 2017. So that's what this function did so I make sure that the names of the files are right. So this. Yeah, it's a general thing in, in I think machine learning to, you know, you have a lot of preparation on your data so this is how we help handle the preparation of your data and keep it kind of comprehensible. So let me do that one more time without. So we have one output. I think so now it's doing a prediction. So took about one second. So here's my output. So I can plot that output with also one of the plotting functions that we provide. So here we see what the classification found based on our training data and the model. Let's see if you can find that one. If it makes sense to us. So here it is in on a map. So here we have the Netherlands we are now right here. So here we are we here. I think this is us, we are here. Right here. Okay, so let's see what what what is it. So we are value zero. Seven. Okay, so we're seven. So let's see what it is. If that makes sense. Pastures. Well, we tried this was our first try. Okay, so let's see what this this is zero. So that's more than Heathland. So this can help you kind of see, you know, what's happening with my model. You have a question. Just about the credit sample. Yeah, we have it. But actually we, we have like, we have like, you know, we have all these package relies on the functionalities of signkit learn inside of this signkit learn. And you'll have a validation strategy that's used like a crew by being to split the data. So you have this C.V. group. And column. So what actually what we do in the, in our production work, we did that. So all the tiles that I presented in the morning. So they, they are like one. And then we have the C.V. group. So you have this C.V. group. And then we have the tiles that I presented in the morning. So they, they are like one 30 kilometers by 30. So we use it like the tile ID as the group column. So to do like this special, this the cross validation, it will be like sort of spatial presentation that will use this ID. So you can avoid, for example, train a model with samples from one tile and validate with samples of the same time. So, and, but it's, it's just like, you need to change the CV to group and fold. So it's, it's a, it's implemented, it is implemented in the site kit learn. What we did was just like a integrated integrated to this functionality. So you can use this one, right? Yes. So you can use this one. So now I use here somewhere. Let's see. You just use the, yeah, because here we are just using, for example, the, it's just one pile. So we could split this point in like soup tiles and create this blocking structure. But for, for this example, probably it will work, but we need to really define this in the data frame. So that's a, that was a good question. Yeah, good question. Thanks. Let's see. Yeah, so in here, I think you can see, yeah, now we select on the tile ID, but you can then use it to do a spatial cross validation. And the next thing is that so all these, all these things you can use them within the package, right? So you can, now we use a K fold, but now we spoke about this group K fold, but you can apply different kinds of strategies here. Does that make sense? Yeah, great. Okay. The models. Let's see. I think. Where are they? Let me look further. This one. Yeah, so you have, because this, this, this libraries is, they create like this base class, this base estimator. So all the algorithms that are implemented here, they have kind of the same methods. So our package actually use all these methods training predict and things like that. So for example, you have like the SVM so if you replace the rental forest for SVM, you will have the same structure that we are showing here, but with a different model, the different learner. I think on top and bottom, you can see that there. So you can use like a linear regression. So we will use more learners, but it's fully compatible what we have here in the. Yeah. And I think you will show an example where we use multiple learners to do a ensemble machine learning. Okay, so we have now looked at this example so our first try we didn't do any hyper parameter optimization we just train a very simple model on our training data. And then now we will try and do some hyper parameter optimization and see if our results are actually better. Yeah, we already saw this one so this allows is also as a learn functionality that we will also use. Then the rest, this is kind of similar to what we did before. But here we define our grid search so we will run over the different parameters that we want to test. We want to test maximum depth and the maximum features and the number of estimators. So we want to test all these combinations of parameters and the rest and we use F one score that's also a standard scoring in in the S K learn package. But here we can also define our own. I think we will do it a bit later. We just input the estimator. And then the only thing we need to do here is just add the high parameter parameter and that will allow us to actually do this. And then we will create search. Let's see if we can do this. So we get the similar outputs. And then now we will train. We will actually train the model 90 times and see which one is best. So that might take a little bit of time. Let's see. What the computer is working on it. Some nice green bars. So it's still running. So now it's almost done. I think it's done. Yeah. So now we can see if the prediction actually better than before. So we have. What is it about a 6% increase, I think, in accuracy. So it's, it's, it's something. It's an increase. So that's good. Then we will do the same. We will, we can plot the. Yeah, the, the methods to do some invitation of our results. And we can look again at our results. We can save the model again and load it again. And I think, did we get the output file. Or not. No, okay, we skip the prediction. Okay, but we can do a prediction model. Well, let's have a look first at the probabilities and then I will, we can have a look at also these files. Now I'm using log loss score. So this is an example of a custom loss function that we make. We also use the escalator and implementation of this, adapted to our, our case. And we do the same thing here, but here we also tell our class that we want probabilities. So because we use this random force with a lot of different predictors we can see if they are corresponding to each other or not and if they are all pretty sure about the class then the probability is high and otherwise, the probability is low, or could be lower. So we make prediction again. And let's see if there's different results. We need to train it of course. So now we use a different loss function so we will. It's a bit different to compare it. So here we have now our accuracy metrics but then get towards the last metric that we defined for each class again. So that's, so we get from the model we get different types of accuracy metrics that we can all access through through these probability metrics variable. And then we can also print them. All right. And then we can see if this loss function actually gives us a better results. By looking at the, by looking at the prediction results. So now I'm applying the same function so to predict with the layers. But here I'm putting all the years. No. So now we're still looking at just 2018. So here's the same function to find the right data and to put all that together. And then we're loading it into the landmapper class and producing a prediction but now not just the class but also the probabilities for the different classes. And we also provide the plot roster function to actually have a look at the results. So, yeah, it's of course, very rough interpretation but if you look at this one compared to the other one. Let's see where it is. You see it's a bit cleaner. It's not sure if that's better but at least you can visually appreciate the difference in the result. So if it's better that's of course matter of further interpretation and looking at the data. But now we have a bit more more at hand to actually do these interpretations. So we can now also have a look at the, at the probabilities. So I think these are related with. So one is related with zero in this case. Yeah, okay. So I'm not trying to visualize the probabilities for the Morrison heat length that we just produced using the landmapper class. So, here we see the probabilities for this area. So again this is where we are now. Let's see. So this is the bargaining area this is bargaining town and here is the conference center. So now I can have a look where it's likely to have more in heat length. Yeah, let's look what's this. So this actually airports. So at least now we can see what what what areas have impact on our, on our predictions. And I should actually what I like to do is, because it's still the easiest way to to act to have compared these things to go to our viewer and see, you can also see the Lancet data over time. So that actually gives us a lot of information on what it is we actually see in our training data directly and then you can. Yeah, it's not. You can probably also load it in GIS but this one is like really, I think really user friendly to go to different years. We have to. Okay. This one. Yeah, yeah. That's it. So here we can have a look again. And now we have layer coniferous forest. So let's have a look. If this is corresponding to what we found. So here we are. It's a tricky class, because it's very much like I think many other classes. But hopefully our model also picks up these other classes and actually produce higher probabilities for these so that even though we see high probability here, maybe other classes still have higher probabilities there. So let's have a look. We now created our own prediction here. Let's see. Let's stay close to where we know where we are. So this patch for instance. Yeah, that's different. So now I'm observing that our predictor kind of finds a patch of heat land here or at least a higher probability than that we have found in the prediction codes that we used for the Yeah, that we used for this. So that's interesting I think you know this is. Yeah, just some interpretation of the results that you can do and I think all these tools they really help you with. Yeah, visually interpreting also your results and see what what you find in your training data. So that's very interesting. Let's see where we're, and you can do this for all these classes so this class five for instance that's going to first forest. So this makes a lot of sense if I look at this because if you are here, Wageningen then this is the veil with that's the biggest force in the Netherlands. So for me, I think okay this is at least there is quite some some truth in this prediction because I know this is forced area. Of course, there are many much more research to be done to confirm this. And then I think the under all you want to show the people the ensemble machine learning. Yeah. Yeah, okay. Thank you. So I'm not sure any questions about this part. Please. No. Okay. Yeah, so. And now we have like the, the probability results. And I would like just to. So, please show how to execute it. And I would like to explain this report here. And the metrics that we generated so. Here you can see that this is, you can see this report because Chris instantiated the landmapper with the verbals through so even for example for for us that we are developing this this API and we are using like on a daily base. It's really important to follow the process so follow how what it part of the, the, how which part of the workflow it's executing. So these verbals it will like explain everything that happens in the landmapper. So, the main, the main points here it's really, you can use all the class that are available in the signkit learn. So, random forest for learners but you can also use SVM you can use a Keras classifier that it's a implementation compatible with signkit learn. So I will use it, but you also can use it individually. And you can use different types of validation strategies. So here we are just using mainly like cross validation so it's, it's, there are a lot of discussion if the cross validation it's, it's the right way to evaluate it. But what we decided when we were implementing the library is inside of the library will mainly use cross validation. So if you want to validate your result you can do it outside so it's just a matter of have independent points and do the predictions using like the predict points method. So, Chris presented the, the interface here so you can see that you have these predict points. So if you have like independent points you can manage it outside. So, and, for example, you here we are using these Lucas points, combined with Korean Korean land cover, and we are creating our data set. And for the production work actually we use it almost 6 million of points so it's much more points that we are presenting here in this training because it here it's just a matter of how we demonstrate these functionalities. But, for example, we have the model train it now our production model like to produce the maps that are available now. And we can conduct, for example, I independent validation strategy and we can spread points all over the Europe and expected it by visual interpretation or even go to the field if we have money and the context of the project, collect the actual class of it and just use these points to do another prediction so it's a completely independent approach that we, it's not really related with the training and how we are implementing these inside of the model inside of the land map. So, you have this, this possibility. But what it's important here is, we are using the cross validation, because it's a kind of, it's a kind of metric that allows you to evaluate your training process, how you are capturing the variability in the class how your features are really like correlated with the, the expected variable and you can use this, like a reliable way to see if the model it's doing a good job, but to provide like a real validation of your result. And you have like independent data set and it's better provide like a proper sampling design so spread points considering like all the assumptions of the statistics and like a random sign or stratified design to do it but of course, sometimes you are limited, you don't have like budget enough to collect points to produce new data. So you need to work just for example with cross validation but in the literature now you have this kind of discussion and it's, and I personally I don't believe that a cross validation it's able to provide like a real and a proper validation for your prediction so it's a nice way to understand if your model it's doing a good job. It's a nice way if you, to understand if you are optimizing it, but to provide the real validation it's like independent data set so it's, it's, it's what we presented in the beginning of the presentation so you actually you need to have no new data so and, and you can try to split it, but for for the context of the land cover it's it's, it's how it's the state of the art, sampling design to provide a better validation and a proper validation for your product. So, you, you have it here, not all these aspects but you have the possibility to use what we train it now to predict new points, not just the, the, the, the, raster raster files. But if you back here. So you can see that considering this cross validation strategy that we are using here it's a five fold cross validation, the landmapper, it's actually dual optimization approach to find what are the best hyper parameters. And it's find the best, it's something like vague. So basically what it's doing, it's calculating some currents metric. And here you can see that Chris explained it. So you have here it's a F1 score weighted and it needs to be weighted because you have, it's a multi-class classification. But for example, if you click here, you can see that all those metrics are really available in the signkit learn. So we could optimize also by the currency or by the precision and by the recall that are actually in the context of remote sensing are like user and producer currency. So it's possible do it. But so it's really like it's a modular approach. So you have like validation strategy defined, you have which parameters that you want to test. So all the parameters that I put here, it will be a combination of all these parameters. And you have different strategies also. So I always prefer to find the parameters because it's sometimes you really need to think about what not all hyper parameters needs to be optimized. And so, and it depends of each algorithm. So define these values actually force you to understand a bit more about these learners, about these models and select actually what are like the best, not the best, but the proper parameters that you need to optimize. So in here, we are optimizing the number of trees. So it's something that it's a common standard in the most part of machine learn applied to land cover classification and other domains. But if you provide like, I don't know, 60, 80 trees, so you should have a good result. So considering like dominant class, hard class. So it's a kind of standard value. But of course, if you put more trees, you will provide, you will have a higher computation time because each tree needs to be computed, need to be trained, need to be predicted. So, and this number, it's actually, it's really high correlated with the performance of the model. So for example, if you have 40 trees and you feel you have 40 course, that's not our case here, definitely, we just have four. But if you have like 40 course, each of these trees will run in parallel. So the process will be way faster. So it's this hyperparameter optimization, you want to optimize to find the best model, but there is a plateau. So you need to be aware that sometimes you are just adding computational time and it will increase, I don't know, 0.5% in your records. But for some applications, maybe this 0.5, it's really important. So if we are doing like with, for example, medical applications, it's life. So you need to optimize at most, but it depends off the way and the domain and the application that you are trying to do. So you have this, for example, flexibility here and even for land cover and for geo-spatial classification, maybe you need to really improve the accuracy at most and process more with a high computational cost. So it really depends. And here you have like these other three parameters that are actually the, what is the maximum depth of the each tree? So if you think that these three will like be trained like you will split the data set on each node of these three and this maximum depth will be actually, the default parameter here, it's non. So it will grows till like separate all the samples in the training set. But sometimes you, if you reduce this depth, you will have like a shorter trees and maybe it could help for the probabilities because you will have like more fuzzy output. And if you combine all these trees, maybe it will help. But it depends off your model and the way that you are training. Not the way that you are training, but your training data set. And the maximum feature, it's actually how much features it will be analyzed for each split. So for example, if you put here 0.5, we have 86 features. So it will use half of it to define the best variable to separate the training samples. So, and for example, sometimes I use it like one here. It's totally random. So the random forest will be trained like for each tree and each node, each part of these three, it will select randomly one variable and it will try like split. And maybe this variable, it's meaningless. So maybe this variable, it's not really performing well to separate your point samples, but it will create this node past to another level and select other variables. So maybe you could have like a really deeper trees, but it could help. So it's difficult to predict it. So you need to really think in some parameters, think about what are your computational capacity and think in how you will do the predictions. Because I don't know, maybe we can have a random forest with 1,000 trees, but how we will run it. We have a server to do it or maybe it could produce a very long computation time. So you need to balance all this to define these grids or CV. So it's why I really like to not use them, there is one strategy that you just put some random value between like one range. But I think it's nice to, if you think about it and really try to put some values that are specific for your domain, for your problem and for your application. So mainly here, what we are seeing is all the so you can see that there are 18 candidates. So it's mainly three multiplied by three. And so yes, three, three whites should be less. So 18, yeah, three. Yeah, three by three, nine. Yeah, multiplied by two, 18. So actually, if you think in all these combinations, you have 18 combinations of hyperparameters. So it grows fast. So I don't know if I put like more two parameters here in each of these options, it will be more than 100 combinations and you can quickly blows the process. Not here in our case because it's a small dataset, but mainly for all these 18 candidates, we are using a five-fold cross validation strategy. And so actually we are training 90 models. So, and here you can see the result of the F1 score, weighted F1 score for each of these combinations. And here we are optimizing it by the higher value. So, and the best parameter is this one. So maybe we could optimize a bit more, but for this dataset and for the random forest, we can put more trees if that was the, maybe that could help, but maybe not too much. So, yeah, I think this, yeah, I think here we stop for a break. And I will back for the second part and we can analyze better the probabilities and predict other years. So, and there is, I don't know if there is questions, there are questions. Nothing to chat. So, okay. So let's have a break and we back in half an hour, right?
explain how to do it specifically in using is machine learning, and if you think in a this data. So if you think like in an image, we use it to predict and to generate predictions. So need to prepare the input data set, you need to to get this, this model and really scale it up aware what you plan to do with the final model. these points over like raster data, different need to train a model, we need to evaluate this hyperparameter optimization. So you can test be publishing on this as well. So that's also do the predictions using like the predict points separate the training samples. So, and for