Time-series analysis of disease data
Dub, Timothee
Hengl, Tomislav
In this session, Timothee Dub (Finnish Institute for Health and Welfare, Finland) & Tom Hengl (OpenGeoHub, Netherlands) discussed the basics of Time Series Analysis, including with panel data. We looked into how to take into account seasonality, how to identify a trend and how to investigate the relationship between two-time series, with a focus on practical tips and R packages. By the end of this lecture, participants are able to analyze surveillance data, identify seasonality and investigate potential trends.
 What I would like us to achieve within BISARW is to understand the relevance of time series analysis of disease data to learn how to identify and take into account periodicity or cycles of seasonality you can use as many different names as you wish, how to identify your trend in your disease data and how to investigate a relationship between two time series we will be very very short on that I think it will be just one slide but that's the simplest stuff once you understood the rest. A disclaimer is that a whole week is not enough so you'll just have one hour and I hope you will get some nice information and enough knowledge to try and get started doing some time series analysis with data that would make sense after this hour and a half I've inserted a few bits and pieces of our code here and there that you can copy paste around but I guess you all know are much better than I do and I've stolen that most of those slides to some colleagues from ECDC so I really acknowledge them here so I had to be a pain in the ass with Willy through that slide and now we get to the real stuff so time series data is data points collected over an interval of time like so several data points at different time points the successive observations are usually not independent if it's 35 degrees today it is very likely that yesterday it was not minus five so those observations are not independent they kind of follow each other each others so what do we use it for in EP we use it for surveillance description of data and reporting to generate some hypothesis regarding seasonality or transmission cycles transmission periods and once again information for action we use it to detect outbreaks and respond early we use it to allocate resources healthcare facilities and stuff we can also use it for forecasting and for evaluating interventions we will not be touching forecasting today so a few examples here the evaluation of an intervention and in 1998 they screen poultry stocks for some salmonella types at regular intervals and if positives they were they would slaughter the whole flock he'd treat egg clean disinfection and of the breeding area that's an intervention which according to our time series analysis was clearly effective on the serotypes that were of salmonella that were targeted as you can see in the stuff below you can also use it for outbreak detection and you're going to use some models some forecasting models with some prediction intervals that will tell you whether you have more cases that you should observe or that you should expect so you can have some threshold for early detection and response and there is an example of an art package which I have never used but it this one was recommended to me you also have other activities like heat wave preparedness that can be conducted with time series but once again it all comes back to the quality of the data that's the surveillance and epidemiologist in me saying that you need to assess whether your data is good if it's representative of your surveillance system the idea like will access to diagnostics will be similar in all the area is the reporting as timely across the whole country I don't know if you've ever used a register like infectious disease register data but you usually have several options for a date you've got the date of sampling you've got the date of notification you've got the date of suspicion well if your register has that stuff I'm interested but usually you can have the date of first consultation for seeking of care you can you can have that in some registers so you need to think about the date you wish to use and this is something to keep in mind especially if some in some areas the reporting can be delayed and there's also something a bit weird and funny is that we're attracted to round numbers in the late 90s in the US they looked at data onset from diseases notified and you have twice more on the first day of each month that's a reporting issue and that's something you will have to think of when working on your analysis with time series data if you're using surveillance data and if the quality is not that great so garbage in garbage out the quality of the data thank the epidemiologist if the data is good thank the data managers who run the surveillance system do not call the surveillance system a cow with makeup some colleagues will let you know about that so how do you want the data that you will be using to be you want it to be aggregated you will need to think about the time units you want the time unit that is of relevance for your work so I like to use month because I usually work on small stuff like tickborn and cephalitis cases in Finland but for influenza transmission or for something like covid weekly basis can be quite interesting you will need to think about your areas also especially if you want to do some panel data analysis and if you're considering using covariates such as weather if you have a very large area you will totally dilute the effect of that far of that parameter so you always have to take everything into consideration there are a few art packages of interest that I've listed here you can play around it during the crash course and I will stay for the crash course maybe depending on my luggage gets here so the time series has four components three and a half you have the trend which is whether an event increase or decreases the frequency of an event decreases or increases over time and then you have what I call what we can call periodicity is whether you're the occurrence of the event follows some cycles cycles are supposed to be defined as above a year and then you have the seasonal component which is what happens during a short period less than a year and then you have the random component this stuff when you will work on your model will most likely be your residuals daily so if we're thinking disease data you will have it would be a c if you're into disease data we're not going to look at the variations within the date which is why I've never prepared for that question so it's it would be a something like a periodicity but I would not call it a cycle for me they're both periods but in AP they like to call seasonal something that's below a year and cycles something that's over several years so if you decompose your time series data it's built up of three things the trend of a time which here is a bit of the influx the periodicity because we're looking I forgot the scale here shame on me because we're dealing with something that's yet that appears to be like a 52 weeks cycle and then you will have your residuals your random component so what's the trend that's a bit too simple for you maybe so it's the long-term direction of the time series you have a trend if there is a long-term increase or decrease your data can also have no trend you can have stuff that's super stable and a trend is not necessarily linear so how will you identify your trend super easy you plot your data how do you analyze it plenty of options we look so it can be a bit complicated if you look for example at weekly cases of legionellos here legionellos is here it can be a bit complicated to see if there is a trend in that situation because it's it's rather flat but if you have a quick and dirty linear regression linear regression you see that you have an upward increase over time and well this was too simple we can skip and that was the same again so still as I was saying when you don't have your linear stuff it's a bit complicated to have a direct idea of what your trend would be so step one you plot your data that's fairly easy I think you all know that already step two you use you can use different techniques to analyze the trend and the easiest would be to smooth to smooth it up using a moving average what's a moving average will be the next question you in EP we use centered moving average so for example here we have monthly cases of the disease you will use here it's a five weeks moving average that you're using so you're centering of the week of interest and take the two weeks before and two weeks after and you do the average of that this will give you something that doesn't have those very very that's those way too many variations this gives you an idea of the trend over time so it also means if you want to investigate a disease over the year 2020 to 2022 when you will ask for the data to your friendly data manager you ask for two months before the beginning of your period of interest after because if you come back to him two weeks later when you want that and if you've called his register a car with makeup he will not be happy so that's a fairly simple example moving average it will not be a problem for you and the wider window you choose the flatter and stable your curve will appear this is just for you to see what's happening for your plotting and you can see that with a 51 weeks window over a period of more than 10 years you see a clear very slight increase over time which might be very difficult to to see with a five weeks window or with that 25 week window are you still following okay good now simpler solution if just like me you can't see anything when you look at those kind of graphs go for a very simple linear regression where you will have well you will just put your time variable so year week if you're looking at something after weekly data or month into your model and then you will have a coefficient for that and this you don't need I think and this yeah so and you will get your coefficient that gives you the coefficient of increase of cases over time I have a personal preference for using negative binomial regression instead of linear because it's less cumbersome regarding assumptions that will just touch it up a bit later then you have the difference between your observed and your predicted your residuals and we'll discuss it a bit later so you're have a few assumptions when using a linear regression that setting and but also when assessing a trend you want the association between the two variables to be linear except that if you have an intervention that occurs at one point you might want to add this into the month into your model and that would be another topic you want the your errors to be independence you want your horrible horrible that you have to respect normality so for each value of x the value of y are normally distributed and that's where I'm a little bit out of my depth being more an epidemiologist who will use and look at the stuff and be like it doesn't look like it's supposed to look and just go ahead and do something else and then you want to have an equal variance over time so here is a simple linear regression where they looked at assessing the trends and your coefficient will be what you're interested in I'm sorry this is a stator output I didn't have time to redo them with r and that coefficient will give you the slope of your trend over time and here you will not look at p values so this is how your simple linear regression actually looks like for that data now we have to talk about the residuals you know probably all more about modeling than I do but your residuals is the difference between what you've observed what you've seen in real life and what your model estimates and you want so much stuff from those residuals you want them to follow normal distribution you want their variance to be around zero and for example this I'm not happy with it so you need to find other options other models that have less assumptions than issue or you can log the stuff to reduce the spread of the residuals and you end up having something that looks way much nicer and that's the type of residuals graph I'm happy with but the problem with that logging is that the interpretation of coefficients will change and it will get a bit more complicated so with each unit increase in the dependent variable you will have a multiplicative percentage change in independent variable seriously log transformation I'd rather go for negative polynomial always so some key ideas in your trend analysis you plot your series and you're moving at you look at the structure you describe it you decide whether a log transformation is needed and you decide not to do it you fit your regression model you interpret you check your residual and make sure you interpret your coefficients correctly the numbers need to make sense in the real world 0.0001 percent increase over time per month or year is not that interesting except if you're working on a million years data okay so next component the cycles and the seasons here you can see in that data that you have some cycles over a period of three to four years this is exactly what you want to look at and take into account when modeling time series data and this is how a model would look like and how we observe with your flag the seasonal variation is within a year and they tend to repeat themselves each year that's Campylobacter for example and you can clearly see it and that's a brand new one that we had a quick and dirty look this week it's chlamydia infections notified in Finland so you can see that COVID restrictions didn't do much to the spread of chlamydia over the past two years if you know anybody who's into zero COVID dogma you can tell them that if well yeah just let them have a look at that and then you will have your random variation which I wanted to talk about a little bit later but so your classic time series is the trend the cycle if there is a cycle if you have long enough period to see that because if you only have two years and if there are cycles that are way larger you won't get them the seasonal periodicity and then the random stuff it's every time series data plot can be decomposed as follows and by adding all that stuff the trend the cycle the noise and everything you might end up having something not too far from the real life and this is an example of something we did for tickborn encephalitis in Finland over 11 years and as you can see we had that trend over time not linear we had those trend plus a 12 month cycles that you can see here and as you can see we are not too bad at predicting the incidence of TBE except we don't really know what happened afterwards in the two years that followed we had way more of an increase and and now my model sucks so how do you assess the seasonality other than by plotting and that's where I've copy pasted some R code the spectral analysis which is a very fancy word to say periodogram is what you want to do first and there is that very no yes so if you have a data set with cases per week weekly cases or monthly cases and but keep in mind that this doesn't work with panel data you need whole aggregated so one observation per period you have that very easy formulas from it's actually from base R that will allow you to plot a periodogram and you will have to decide on a period so here for example we decided that we wanted the scale of the periodogram to be 52 weeks and if you look at that I have that periodogram with two peaks plus a third one those two peaks are the main period which is actually because we decided of the 52 week scale 52 weeks divided by one so a 52 week period and a 26 weeks 52 weeks divided by two another 26 week period that you get in there and that's when you start with the fun stuff because every curve that's that is a cycle can be decomposed in sine and cosine so when you want to add a periodicity or a seasonality to your model then you will just generate a sin a sinus and a cosine of those periods key message you plot your series you detect and describe a pattern you play around with sine and cosine then you interpret your coefficients in terms of something that has to make sense those 0.0.0.0.1 increase per 10 years you're not really interested try to keep it simple and then check your residuals but then because of all that crap with residuals that always make me very unhappy you have another possibility you can consider other models that linear regression and I would say that I have a strong preference for negative binomial regression because you have less assumptions to bother with and more veteran statisticians would tell you that now that we have the computer power to run them quite easily on almost any laptop it's quite easy to go straight to that and it allows epidemiologists to pretend to know what they're doing which I really like so what I will talk about also time series but and now it's a space time so it's a time series but we work basically with the both space and time and there are basically two versions of analysis can do when you have both space and time you can do simultaneous analysis of trends so one of the option is this spar package in R you can do you can feed this space time densities and but this will only work with the current only data so no quantities and then the second option is to first harmonize everything in space and then do the time and so I'm going to show you well tomorrow I will show you the spar a bit but our main focus is on this number two so the version number two which is first harmonize the space then analyze time series and then you focus only on time and then what we do basically we apply type of modeling that you saw from team we applied for every pixel so it becomes very very computational okay super computational and also the what's important to discuss is this so there's the seasonal components there's the cyclical components I'm not going to talk about it for me it's all kind of this sinusoidal or whatever curves then there's also breakpoints I think you you don't have that maybe in the health data but in vegetation data there's a breakpoints like when there's a fire or natural hazard and then also there are the distributions they also change and so that's also important when you run the analysis and let me see this is from the Wikipedia by the way these things from Wikipedia if you if you open it in a let's say so if you open this in Wikipedia you go to this image it's as usual in Wikipedia if you look at the meta information this looks like some real data set right no it's not a real data set it is let's see more details it's been generated through our code it's a simulated data so it's been generated it's a simple it's a it's a dummy data and they they played a bit so they wanted to show so if you use the raw data the simple thread line it's almost worthless so it's like r square 0.0 0.08 right but if you do the filtering then the model is 0 r square is 0.97 so the same basically same data set but you see there's a huge difference how you implement modeling going from no correlation to to low r square to high r square so yeah you can take a look at that but that's a classical data set and this thing also T mentioned there are these components so this will be a original signal and you can extract the seasonality and then you can maybe detect the trend and then the remaining things is the random component and random component is noise you don't want to model never fit a model to noise please that's a physics that's not statistics but yes seasonality and trend that's something we're interested and so as I said the team also mentioned that I was I will still say this noise noisy component yeah be very careful and I only put here one so I cyclic hollow seasonality I put it together but I agree there's more components here so number two there's a ABC I don't know there's more versions this is the global temperature measures for the last 10 000 years and also you see this I think uncertainty the the blue line is uncertainty and so what you see is how how the scale that you look at you know like a annual temperature the scale that you look at actually the world's been warming up a bit I think since the glacier period 10 000 years ago there was a cold cold period here and then the the world's been warming up and it's been actually cooling down it's been cooling down but now we have industrial evolution and it's the red line and you see it's a one degree and it's completely off the scale it's very difficult to even visualize it it's a really hockey stick but so that depends also this time series analysis depends how far you look at okay now the space time as I said we do the we interest in space time analysis of trends and so now we have one data set my colleague Leandro here he's been running this analysis and we downloaded all of the modest data set actually it's almost 14 terabyte of data so we downloaded and we took this data set and we prepare like a data cube and then we run the time series analysis because we would like to see where is the the NDVI you know the normalize the difference vegetation index it's like a vegetation index it's a measure of primary productivity of landscape we would like to know where is the negative trend in NDVI where is the positive because that's the measure of land degradation let's say so we wanted to analyze that and we prepare like these data cubes as I said that's really can be really large especially if you do global we also prepare for Europe at 30 meter we have global at 250 meter and then we want to estimate this trend and then the first thing we realized that there's many missing values for example there's a lot of lot of missing values then also there's artifacts there's some outliers and and then okay we have to do some gap filling and so after we remove the outliers and we do gap filling then we can do trend analysis and as a result of trend analysis this is just fitting a linear model to data we can get the map of the slope this is the alpha the intercept so that's the beginning temperature the beginning NDVI at 2000 and then the slope it's a positive or negative negative means decrease of NDVI and decrease and positive red color here it means increase but as I said in between we have to we have to derive all this time series again and we have to actually convert all the layers first to make to fill in the gaps after we fill in the gaps then we remove the trend or the seasonality component because the NDVI changes you know winter summer for every pixel winter summer is a bit different right in the world and then once we remove this component then we can calculate per pixel and imagine this is a huge image it's like 150 000 by I don't know 100 000 pixels they're huge images and we take the time series of pixels and we run models through time for every pixel and we have to run it in parallel and it will take like a one day of computing with full capacity right Landro this NDVI will be like a one day and then so this is one this is one row data this is a big monthly image for for January and you see there's a lot of missing pixels if we will like to compute any trends on this data it will not be possible because we have too many missing pixels so we have to do this gap filling and then after gap filling trend deep trending and then the trend analysis so this is the one image this is our computing so this was with the 400 threads we have not 1000 threads so we can speed it up a bit but this is you see running on on servers processing in parallel every pixel time series running in parallel and this is the NDVI after gap filling you can see how the world our world breathes basically right so this is the vegetation changing Landro made this realization it's really mind-blowing right many people interested in that and it's nice to see it after you remove all the gaps you know and everything so it's nice to see it but you see there's a seasonality effect so if we will fit a trend line it will be difficult like in this first example to show this Wikipedia example it will be difficult to fit a trend line so we do this is one point with the NDVI this is the original NDVI this is the seasonality and then this is after removal after removal you see we have just the trend and these are the residuals of the of the model and so here we estimate that there's an increase in NDVI for this pixel in the last last 20 years there's an increase in NDVI so it could be that they started planting reforesting I don't know and this is the same image of the world the same the same months but after you use remove the seasonality do you see a difference I mean if you watch carefully you can see that there are some differences and these could be differences for example due to El Nino La Nina effects you know it could be also differences because of land degradation so there's a there's a multiple not things but this is now the seasonality is taken out so you can see now also a feature which is basically something that the team explained only temporarily now we looked at it in the space we look at the space what happens with vegetation last 20 years taking out the seasonality and then we can compare of course the beginning and the end NDVI that's also one way to detect but this wouldn't wouldn't work because by accident you could have a difference between begin and end by by accident could be small difference but through the time it's actually bigger and then this is the result ordinary least square fitted for every pixel red color red color means it's a decrease in NDVI a green color it's an increase and so what happens what is the conclusion what do you see immediately the world is getting greener and it's been published in a big journals and and the world is getting greener because of actually two countries China and India exactly so you see there's we're getting greener Europe is kind of let's say neutral but there is Eastern Europe is also looks like it's getting greener but there are places where there's reddish pinkish and you see there is some spots where obviously the NDVI has been dropping this is for every pixel and fit models you can also very cool you can save the R square for every pixel so this is R square of the trend analysis and now we can mask out only show only the pixels where the R square is higher than 0.25 so it means there's a not only there's a trend we fitted but this trend is really let's say significant and now you see there's many pixels drop out and then this one is if you take the probability for beta to be significant so less than zero so we can compute all these images actually Landro computes usually from this trend analysis like 16 images can be different components of coefficients of I don't know some if you want to this periodicity you can compute these images and visualize them so yeah and then we can zoom in we can zoom in somewhere in France I think Landro found two spots one spot is highly reddish and the other spots highly green I don't know where these areas are now but you can see when you zoom in on the NDVI data it is clearly this greenish there's a clearly increase in NDVI and here clearly there's a decrease and I think this this area here it's close to some place I'm not sure so you we will have to check but it could be that there were forest plantations and then they cut them down or something so there was a clear cut but in Brazil also you know there's a lot of conversion of land into agriculture if you zoom in also you can clearly see that places where they convert forests to agriculture that you lose NDVI and so we analyze this data and this data is available and you can see that the cities as they grow they cause NDVI to drop of course and so it matches lots of things matches what we expect so sorry can I just start off why would India or China get greener? Because they intensify in agriculture they put more fertilizer in the soil and they had to produce more food and they get greener the question is why did China and India why the greener is that correct Leandro right? It's more intensive production so they have also they rotate through year you know you can do two cropping season or two harvest or three harvest season so they intensify in production this is what we did for the NDVI then we said okay let's play let's do the same thing for the temperatures so exactly the same you remove the seasonality you fill in the gaps you remove seasonality and we did the same thing and people went nuts about this and again you can see we can do it for the daytime temperature nighttime temperature and you can see that the China India is getting cooler because they may get more vegetation so daytime temperature get cooler but the nighttime temperature are more neutral so they're not they're not changing so much but in general you can see here what you see as a trend blue is a negative red is positive what is your first impression? There is a global warming obviously there's more red than blue and where does the global warming happens the most? It looks like it's here Russia and Alaska and all the places where there's for example if you look here this reddish areas this is a change in temperature because they cut the forest and they put the cropland and you see there's all this basically kind of a front line of the conversion of the land all this front line you see reddish so there's a really change a change in temperature it's also interesting to see difference between a big difference between daytime and nighttime temperature and this is a zoom in on Montpellier I have this map I will show you that tomorrow you can access it and you can see the red is red and blue again and you can see there's a in general most of France is the same temperature but somehow this place here for example where they had these plantations of eucalyptus or something there's a wetlands here I think this is the French Basque country and looks like there's an increase in temperature for some reason also the mountains are warmer Pyrenees seem to be warmer and all this area here seems to be getting warmer but gently not not a huge increase okay tiger mosquito so tomorrow I will show you now this was a environmental data time series so tomorrow I will show you the whole case study we get the G-BIF data on mosquitoes I think we start with like 50 000 records of these mosquitoes from G-BIF then we subset to Europe then we finish with 28 000 I think and so we have 28 000 points it's a significant data set for each point we know the the date the year and we know the individual counts so they even have the counts of mosquitoes and that's the stuff I put also in this dashboard that Francesca was showing and so what we do then we analyze this mosquito spread using a space time machine learning so we do this model B we produce the time series of images of occurrences of mosquito and once we produce these images of occurrences then we can visualize them and you see this is now different years going from 2000 to 2021 and you can see that it is spreading the mosquito is slowly spreading towards the inside I'm just showing Spain but so we get that red color is of course high probability of occurrence and a yellowish color is slow and so you can see it spreading yes so I can show you that's the thing I will I will talk to more about I can show us today so this is all in this this tutorial and so we well it's tricky we had to generate this pseudo absences about 10 percent pseudo absences and once we create pseudo absences then we can this is the pseudo absences we generate them by using the max like package and it is highly the mosquito is climatologically it follows specific areas very clearly so you can generate this pseudo absences these are the pseudo absences and then when we do the modeling when we do the modeling then we take about 45 covariates so we have the temperature rainfall nightlight images human impact yeah lots of lots of images snow etc and then when we fit the model the model is significant and what comes up is the most significant covariates actually night lights night lights time series of night lights and then travel time to cities and ports that comes on the top of the list and then once we make that then we can do predictions and we produce predictions based on sorry question so the occurrence only data that's what the biologist like the most it's basically this is the occurrence only data right basically it's fine except you cannot do any machine learning with it I mean I as least I don't know I don't know which which what would you use so because the anything in machine learning it's like either classification regression survival problems if you look at for example classification problems you cannot have only one state you have to have at least two states to start or multiple states so you have like a zero and one state you know where it comes and where it doesn't come so with this thing you only know where it comes but you don't know where it doesn't come so the machine learning cannot train cannot be trained so we need to insert pseudo absences and the pseudo absences there's lots of methods in the literature and most of them they they suggest to be very conservative so only insert for example 10 so small amount of points and to locate it very conservatively in places that you know the mosquito doesn't come one example is I don't know top of the alps right minus 20 degrees I don't know then you know places like Iceland I don't know so you say these are the pseudo absences and then model can at least know where it is absolutely not coming and then once you make a map you get you get this you get these predictions and then it's fine then you can because you can introduce the bias you can introduce if you the pseudo absences are you know they're difficult to validate and we do them only to kind of bridge that gap so we can do machine learning so and most of literature suggests to do to be conservative and limit to like say 10 10 percent maybe if you do 20 percent I mean that's not a strict number but if you look at the the literature so here's the this is the key reference if you look at this literature you can read about it but number one they in this paper they they prove that if you do good quality pseudo absences that it helps with modeling that's number one but they also say you should limit to limit the pseudo absence only to something which is you're highly certain right because imagine if I put a pseudo absence and let's say the mosquito does appear then I confuse my model and I introduce a bias so but these are the pseudo do you have experience somebody has experienced pseudo absences yes yes okay you can do hundred you could do hundred pseudo absences it for machine learning would be enough I hear I do about 10 percent and it's a it's a really no problem I get my you know I can see with the models when we do modeling for these mosquitoes we got really distinct models and a good match you know we have a probability space we have 0.1 0.1 errors plus minus so that's a relatively good match you know you cannot put it in our square because it's a binomial variable but when you when you look at the maps you can see it's really distinct patterns and it's a it matches the the point data very well and we have the covariates you know the the covariates we get also very clearly night light images the then the travel time to cities you see night light images travel time to cities and so it kind of matches what we will expect yes I have 150 I have 150 covariates I have 150 but it's computational to do selection so we went and we went to just put 45 just for the exercise but there's a lot of covariates and covariates are prepared as cloud authors you this I will show you you can connect you can also do your own analysis for them you don't have to download them so I will show you they are available as a geospatial database about 25 gigabytes of data we can put more if you need and they cover a time series from so from 2000 to 2021 but we also have from 1985 we have the data so if somebody wants to go back in the time it's also possible yes this one yes I think the ability of the satellite is by your that could have affected so the about the NDVI analysis Landro was there any problems with the modus in a sense that the images maybe make a bias so maybe the NDVI it's overestimated because of modus sensors it's the same sensor now yes yeah modus it's a high quality product so it's the same sensor running since 2000 so and it's like there are several reprocessing steps to like remove the effect of the atmosphere effect so this NDVI was based in what we call like a surface reflectance so it of course it tries to simulate how would be the response of the land and conceiving like a sensor close to the surface so I don't see like bias in this product but that's a very nice question because this product is the collection sticks so it's running for for several years now and there is a paper that actually explained that there was a significant difference between the collection five and the collection six so and they fix it in the collection six so you couldn't see this kind of pattern in the collection five so because every time that you have a new collection of the modus product they reprocess the whole archive so and they are doing it for quite some years now so but in the collection six as far as I know it's a high quality data so the answer to that question then to Frank is that most likely these NDVI trends are correct and they do match the literature also they match also LAMSAT data NDVI etc so I paper that you mentioned they actually analyze about three satellites I think they also consider in spots and the probe of the so when all the three satellites are presented a greener trend for the work and then there's a question about the was there any ground routing so yes for NDVI it's there is for example extreme events they are documented for example fires clear cuts of forest they're documented there let's say you don't have to go on the field you know you can find this fire fire data sets and clear cuts and then you can compare and you can see that you have a negative NDVI right the way you have a fire and or like a clear cut you will have a negative negative beta in NDVI so yes there is ground routing not it sends that you go on the ground but you could get for example high much high resolution images up to today you can get up to 30 centimeters satellite images did you know that 30 centimeters and then with that you could see that there is a really somebody cut the forest but there was a forest fire or landslide or flood and then the NDVI drops please send us the link to the covariate sources no problem the covariate sources there is a CSV file there's the CSV file I just have to open it in the github so under inputs and then CSV somewhere not here mood layers one kilometer all the layers are perfectly stacked they're all 100 percent gap field and they're all perfectly stacked and they're all cloud opt-in geotips and you can just grab them from here you can open them and put them and you can see the source ID you can see what what are they what do they mean and you can read usually they're all published in the publications at the moment there is about 500 almost 450 layers 450 but as I said we have much more it's just I didn't want to overload you I mean we could put one and a half thousand layers probably easily or two thousand layers and then these are all cloud opt-in geotips running on S3 so you don't have to download it you can just make in a code connection and then you can do extract crop overlay extract so you can do you can manipulate and it will only pick up the it's like SQL query it will only pick up the data it needs it's a proper data cube so okay this was my little contributions to time series analysis we look forward to showing you how to do this in an hour and I have also I put actually two ways you can do this trend maps one way is to use a package that will do it so there is the trend roster function in the green green brown package and so is anybody using that package maybe it's for the for the images time time series analysis image you can detect break lines you can set up the seasonality effects and you can say that if you know for example if there's like 10 break lines you know you can you can specify them so that's the package sorry break line is let's say when you have a graph change in values in the time series data so that will be a clear cut of a forest or like a fire or flood yes what they say break point yeah so that's the green green brown package but it doesn't run in parallel and you can run it it's also a bit trouble to install it it's not on it's not on cron anymore I think so it's a bit more trouble to to make it so what I did here this is horrible code but look at this this is running linear modeling in parallel for every pixel so you make a data cube and you have time series like let's say 30 years I don't know and then you can fit any model you like for every time series and it will run in parallel so I analyze here like 1 million pixels something like that maybe with less analyze it in 20 seconds with with 32 threads so in 20 seconds poof done if I had more trends it's two seconds done and this is the code to do parallelization but it's a bit abstract it's you know and you know and when parallelization there's yeah there's steps you have to it's a bit abstract so but the the important thing it's in our it's possible to crunch really large datasets and you get this code is fully scalable I can apply it to any type of problem I have to do time series analysis of you know data stacks so I could put also different models maybe something that team was showing these models with this seasonality conversion I don't know so I could put a lot of stuff to model okay so yeah that's for me
 What I would like us to achieve within BISARW is to understand the relevance of time series  analysis of disease data to learn how to identify and take into account periodicity or cycles  of seasonality you can use as many different names as you wish, how to identify your trend  in your disease data and how to investigate a relationship between two time series we  will be very very short on that I think it will be just one slide but that's the simplest  stuff once you understood the rest.  A disclaimer is that a whole week is not enough so you'll just have one hour and I  hope you will get some nice information and enough knowledge to try and get started doing  some time series analysis with data that would make sense after this hour and a half I've  inserted a few bits and pieces of our code here and there that you can copy paste around  but I guess you all know are much better than I do and I've stolen that most of those slides  to some colleagues from ECDC so I really acknowledge them here so I had to be a pain in the ass  with Willy through that slide and now we get to the real stuff so time series data is data  points collected over an interval of time like so several data points at different time points  the successive observations are usually not independent if it's 35 degrees today it is  very likely that yesterday it was not minus five so those observations are not independent  they kind of follow each other each others so what do we use it for in EP we use it for  surveillance description of data and reporting to generate some hypothesis regarding seasonality  or transmission cycles transmission periods and once again information for action we use  it to detect outbreaks and respond early we use it to allocate resources healthcare facilities  and stuff we can also use it for forecasting and for evaluating interventions we will not  be touching forecasting today so a few examples here the evaluation of an intervention and in 1998  they screen poultry stocks for some salmonella types at regular intervals and if positives  they were they would slaughter the whole flock he'd treat egg clean disinfection and of the  breeding area that's an intervention which according to our time series analysis was clearly  effective on the serotypes that were of salmonella that were targeted as you can see in the stuff below  you can also use it for outbreak detection and you're going to use some models some forecasting  models with some prediction intervals that will tell you whether you have more cases that you  should observe or that you should expect so you can have some threshold for early detection  and response and there is an example of an art package which I have never used but it this one  was recommended to me you also have other activities like heat wave preparedness that can be  conducted with time series but once again it all comes back to the quality of the data that's the  surveillance and epidemiologist in me saying that you need to assess whether your data is good if  it's representative of your surveillance system the idea like will access to diagnostics will be  similar in all the area is the reporting as timely across the whole country I don't know if you've  ever used a register like infectious disease register data but you usually have several  options for a date you've got the date of sampling you've got the date of notification you've got the  date of suspicion well if your register has that stuff I'm interested but usually you can have the  date of first consultation for seeking of care you can you can have that in some registers so you  need to think about the date you wish to use and this is something to keep in mind especially if  some in some areas the reporting can be delayed and there's also something a bit weird and funny  is that we're attracted to round numbers in the late 90s in the US they looked at data  onset from diseases notified and you have twice more on the first day of each month  that's a reporting issue and that's something you will have to think of when working on your  analysis with time series data if you're using surveillance data and if the quality is not that  great so garbage in garbage out the quality of the data thank the epidemiologist if the data is  good thank the data managers who run the surveillance system do not call the surveillance system a cow  with makeup some colleagues will let you know about that so how do you want the data that you  will be using to be you want it to be aggregated you will need to think about the time units you want  the time unit that is of relevance for your work so I like to use month because I usually work on  small stuff like tickborn and cephalitis cases in Finland but for influenza transmission or for  something like covid weekly basis can be quite interesting you will need to think about your  areas also especially if you want to do some panel data analysis and if you're considering using  covariates such as weather if you have a very large area you will totally dilute the effect of that  far of that parameter so you always have to take everything into consideration there are a few  art packages of interest that I've listed here you can play around it during the crash course and I  will stay for the crash course maybe depending on my luggage gets here so the time series has four  components three and a half you have the trend which is whether an event increase or decreases  the frequency of an event decreases or increases over time and then you have what I call what we  can call periodicity is whether you're the occurrence of the event follows some cycles  cycles are supposed to be defined as above a year and then you have the seasonal component  which is what happens during a short period less than a year and then you have the random component  this stuff when you will work on your model will most likely be your residuals daily so if we're  thinking disease data you will have it would be a c if you're into disease data we're not going to  look at the variations within the date which is why I've never prepared for that question so it's  it would be a something like a periodicity but I would not call it a cycle for me they're both  periods but in AP they like to call seasonal something that's below a year and cycles something  that's over several years so if you decompose your time series data it's built up of three things  the trend of a time which here is a bit of the influx the periodicity because we're looking  I forgot the scale here shame on me because we're dealing with something that's yet that  appears to be like a 52 weeks cycle and then you will have your residuals your random component  so what's the trend that's a bit too simple for you maybe so it's the long-term direction of the  time series you have a trend if there is a long-term increase or decrease your data can also have no  trend you can have stuff that's super stable and a trend is not necessarily linear so how will you  identify your trend super easy you plot your data how do you analyze it plenty of options we look  so it can be a bit complicated if you look for example at weekly cases of legionellos here  legionellos is here it can be a bit complicated to see if there is a trend in that situation  because it's it's rather flat but if you have a quick and dirty linear regression linear regression  you see that you have an upward increase over time and well this was too simple we can skip  and that was the same again so still as I was saying when you don't have your linear stuff  it's a bit complicated to have a direct idea of what your trend would be so step one you plot your  data that's fairly easy I think you all know that already step two you use you can use different  techniques to analyze the trend and the easiest would be to smooth to smooth it up using a moving  average what's a moving average will be the next question you in EP we use centered moving average  so for example here we have monthly cases of the disease you will use here it's a five weeks moving  average that you're using so you're centering of the week of interest and take the two weeks before  and two weeks after and you do the average of that this will give you something that doesn't have  those very very that's those way too many variations this gives you an idea of the trend over time  so it also means if you want to investigate a disease over the year 2020 to 2022 when you will  ask for the data to your friendly data manager you ask for two months before the beginning  of your period of interest after because if you come back to him two weeks later when you want  that and if you've called his register a car with makeup he will not be happy so that's a fairly  simple example moving average it will not be a problem for you and the wider window you choose  the flatter and stable your curve will appear this is just for you to see what's happening for  your plotting and you can see that with a 51 weeks window over a period of more than 10 years  you see a clear very slight increase over time which might be very difficult to to see with a  five weeks window or with that 25 week window are you still following okay good  now simpler solution if just like me you can't see anything when you look at those kind of graphs  go for a very simple linear regression where you will have well you will just put your  time variable so year week if you're looking at something after weekly data or month into your  model and then you will have a coefficient for that and this you don't need I think  and this yeah so and you will get your coefficient that gives you the coefficient of increase of  cases over time I have a personal preference for using negative binomial regression instead of linear  because it's less cumbersome regarding assumptions that will just touch it up a bit later  then you have the difference between your observed and your predicted your residuals and  we'll discuss it a bit later so you're have a few assumptions when using a linear regression  that setting and but also when assessing a trend you want the association between the two variables  to be linear except that if you have an intervention that occurs at one point you might want to add this  into the month into your model and that would be another topic you want the  your errors to be independence you want your horrible horrible that you have to respect  normality so for each value of x the value of y are normally distributed and that's where I'm a  little bit out of my depth being more an epidemiologist who will use and look at the stuff and be like  it doesn't look like it's supposed to look and just go ahead and do something else and then you  want to have an equal variance over time so here is a simple linear regression where they looked at  assessing the trends and your coefficient will be what you're interested in I'm sorry this is a  stator output I didn't have time to redo them with r and that coefficient will give you the slope  of your trend over time and here you will not look at p values  so this is how your simple linear regression actually looks like for that data  now we have to talk about the residuals you know probably all more about modeling than I do but  your residuals is the difference between what you've observed what you've seen in real life  and what your model estimates and you want so much stuff from those residuals you want them to  follow normal distribution you want their variance to be around zero and for example this I'm not  happy with it so you need to find other options other models that have less assumptions than issue  or you can log the stuff to reduce the spread of the residuals and you end up having something  that looks way much nicer and that's the type of residuals graph I'm happy with  but the problem with that logging is that the interpretation of coefficients will change and  it will get a bit more complicated so with each unit increase in the dependent variable you will  have a multiplicative percentage change in independent variable seriously log transformation  I'd rather go for negative polynomial always  so some key ideas in your trend analysis you plot your series and you're moving at  you look at the structure you describe it you decide whether a log transformation is needed  and you decide not to do it you fit your regression model you interpret you check your residual  and make sure you interpret your coefficients correctly the numbers need to make sense in  the real world 0.0001 percent increase over time per month or year is not that interesting  except if you're working on a million years data  okay  so next component the cycles and the seasons  here you can see in that data that you have some cycles over a period of three to four years  this is exactly what you want to look at and take into account when modeling time series data  and this is how a model would look like and how we observe with your flag  the seasonal variation is within a year and they tend to repeat themselves each year that's  Campylobacter for example and you can clearly see it and that's a brand new one that we had  a quick and dirty look this week it's chlamydia infections notified in Finland so you can see that  COVID restrictions didn't do much to the spread of chlamydia over the past two years  if you know anybody who's into zero COVID dogma you can tell them that if well yeah just let them  have a look at that and then you will have your random variation which I wanted to talk about  a little bit later but so your classic time series is the trend the cycle if there is a  cycle if you have long enough period to see that because if you only have two years and if there  are cycles that are way larger you won't get them the seasonal periodicity and then the random stuff  it's every time series data plot can be decomposed as follows  and by adding all that stuff the trend the cycle the noise and everything you might end up having  something not too far from the real life and this is an example of something we did for  tickborn encephalitis in Finland over 11 years and as you can see we had that trend over time  not linear we had those trend plus a 12 month cycles that you can see here and as you can see  we are not too bad at predicting the incidence of TBE except we don't really know what happened  afterwards in the two years that followed we had way more of an increase and and now my model sucks  so how do you assess the seasonality other than by plotting and that's where I've copy pasted  some R code the spectral analysis which is a very fancy word to say periodogram  is what you want to do first and there is that very no yes so if you have a data set with  cases per week weekly cases or monthly cases and but keep in mind that this doesn't work with panel  data you need whole aggregated so one observation per period you have that very easy formulas from  it's actually from base R that will allow you to plot a periodogram and you will have to decide on  a period so here for example we decided that we wanted the scale of the periodogram to be 52 weeks  and if you look at that I have that periodogram with two peaks plus a third one those two peaks  are the main period which is actually because we decided of the 52 week scale 52 weeks divided by one  so a 52 week period and a 26 weeks 52 weeks divided by two another 26 week period that you get in there  and that's when you start with the fun stuff because every curve that's that is a cycle can be  decomposed in sine and cosine so when you want to add a periodicity or a seasonality to your model  then you will just generate a sin a sinus and a cosine of those periods key message you plot your  series you detect and describe a pattern you play around with sine and cosine then you interpret  your coefficients in terms of something that has to make sense those 0.0.0.0.1 increase  per 10 years you're not really interested try to keep it simple and then check your residuals but then  because of all that crap with residuals that always make me very unhappy you have another  possibility you can consider other models that linear regression and I would say that I have a  strong preference for negative binomial regression because you have less assumptions to bother with  and more veteran statisticians would tell you that now that we have the computer power to  run them quite easily on almost any laptop it's quite easy to go straight to that and  it allows epidemiologists to pretend to know what they're doing which I really like  so what I will talk about also time series but and now it's a space time so it's a time series but  we work basically with the both space and time and there are basically two versions of analysis  can do when you have both space and time you can do simultaneous analysis of trends so one of the  option is this spar package in R you can do you can feed this space time densities  and but this will only work with the current only data so no quantities  and then the second option is to first harmonize everything in space and then do the time and so  I'm going to show you well tomorrow I will show you the spar a bit but our main focus is on this  number two so the version number two which is first harmonize the space then analyze time series  and then you focus only on time and then what we do basically we apply  type of modeling that you saw from team we applied for every pixel so it becomes very very  computational okay super computational and also the what's important to discuss is this so there's  the seasonal components there's the cyclical components I'm not going to talk about it for me  it's all kind of this sinusoidal or whatever curves then there's also breakpoints I think you  you don't have that maybe in the health data but in vegetation data there's a breakpoints  like when there's a fire or natural hazard and then also there are the distributions they also  change and so that's also important when you run the analysis and let me see this is from the  Wikipedia by the way these things from Wikipedia if you if you open it in a let's say  so if you open this in Wikipedia you go to this image it's as usual in Wikipedia if you look  at the meta information this looks like some real data set right no it's not a real data set it is  let's see more details it's been generated through our code it's a simulated data  so it's been generated it's a simple it's a it's a dummy data and they they played a bit so they  wanted to show so if you use the raw data the simple thread line it's almost worthless so it's  like r square 0.0 0.08 right but if you do the filtering then the model is 0 r square is 0.97  so the same basically same data set but you see there's a huge difference how you implement modeling  going from no correlation to to low r square to high r square so yeah you can take a look at that  but that's a classical data set and this thing also T mentioned there are these components so this  will be a original signal and you can extract the seasonality and then you can maybe detect the trend  and then the remaining things is the random component and random component is noise you don't  want to model never fit a model to noise please that's a physics that's not statistics  but yes seasonality and trend that's something we're interested and so as I said the team also  mentioned that I was I will still say this noise noisy component yeah be very careful and I only  put here one so I cyclic hollow seasonality I put it together but I agree there's more components  here so number two there's a ABC I don't know there's more versions this is the global temperature  measures for the last 10 000 years and also you see this I think uncertainty the the blue line is  uncertainty and so what you see is how how the scale that you look at you know like a annual  temperature the scale that you look at actually the world's been warming up a bit I think since the  glacier period 10 000 years ago there was a cold cold period here and then the the world's been  warming up and it's been actually cooling down it's been cooling down but now we have industrial  evolution and it's the red line and you see it's a one degree and it's completely off the scale  it's very difficult to even visualize it it's a really hockey stick but so that depends also this  time series analysis depends how far you look at okay now the space time as I said we do the we  interest in space time analysis of trends and so now we have one data set my colleague Leandro  here he's been running this analysis and we downloaded all of the modest data set actually  it's almost 14 terabyte of data so we downloaded and we took this data set and we prepare like a  data cube and then we run the time series analysis because we would like to see where is the the  NDVI you know the normalize the difference vegetation index it's like a vegetation index  it's a measure of primary productivity of landscape we would like to know where is the negative  trend in NDVI where is the positive because that's the measure of land degradation let's say  so we wanted to analyze that and we prepare like these data cubes as I said that's really  can be really large especially if you do global we also prepare for Europe at 30 meter we have  global at 250 meter and then we want to estimate this trend and then the first thing we realized  that there's many missing values for example there's a lot of lot of missing values then also  there's artifacts there's some outliers and and then okay we have to do some gap filling and so  after we remove the outliers and we do gap filling then we can do trend analysis and as a result  of trend analysis this is just fitting a linear model to data we can get the map of the slope  this is the alpha the intercept so that's the beginning temperature the beginning NDVI at  2000 and then the slope it's a positive or negative negative means decrease of NDVI and  decrease and positive red color here it means increase but as I said in between we have to  we have to derive all this time series again and we have to actually convert all the layers first  to make to fill in the gaps after we fill in the gaps then we remove the trend or the seasonality  component because the NDVI changes you know winter summer for every pixel winter summer is a bit  different right in the world and then once we remove this component then we can calculate per  pixel and imagine this is a huge image it's like 150 000 by I don't know 100 000 pixels they're huge  images and we take the time series of pixels and we run models through time for every pixel  and we have to run it in parallel and it will take like a one day of computing with full capacity  right Landro this NDVI will be like a one day and then so this is one this is one row data  this is a big monthly image for for January and you see there's a lot of missing pixels if we  will like to compute any trends on this data it will not be possible because we have too many  missing pixels so we have to do this gap filling and then after gap filling trend deep trending  and then the trend analysis so this is the one image this is our computing so this was with the  400 threads we have not 1000 threads so we can speed it up a bit but this is you see running on  on servers processing in parallel every pixel time series running in parallel and this is the  NDVI after gap filling you can see how the world our world breathes basically right so this is the  vegetation changing Landro made this realization it's really mind-blowing right many people  interested in that and it's nice to see it after you remove all the gaps you know and everything  so it's nice to see it but you see there's a seasonality effect so if we will fit a trend line  it will be difficult like in this first example to show this Wikipedia example it will be difficult  to fit a trend line so we do this is one point with the NDVI this is the original NDVI this is  the seasonality and then this is after removal after removal you see we have just the trend  and these are the residuals of the of the model and so here we estimate that there's an increase in  NDVI for this pixel in the last last 20 years there's an increase in NDVI so it could be that  they started planting reforesting I don't know and this is the same image of the world the same  the same months but after you use remove the seasonality do you see a difference I mean if you  watch carefully you can see that there are some differences and these could be differences for  example due to El Nino La Nina effects you know it could be also differences because of land  degradation so there's a there's a multiple not things but this is now the seasonality is taken  out so you can see now also a feature which is basically something that the team explained only  temporarily now we looked at it in the space we look at the space what happens with vegetation  last 20 years taking out the seasonality and then we can compare of course the beginning and the  end NDVI that's also one way to detect but this wouldn't wouldn't work because by accident you  could have a difference between begin and end by by accident could be small difference but  through the time it's actually bigger and then this is the result ordinary least square  fitted for every pixel red color red color means it's a decrease in NDVI a green color  it's an increase and so what happens what is the conclusion what do you see immediately  the world is getting greener and it's been published in a big journals and and the world  is getting greener because of actually two countries China and India exactly so you see  there's we're getting greener Europe is kind of let's say neutral but there is Eastern Europe is  also looks like it's getting greener but there are places where there's reddish pinkish and you see  there is some spots where obviously the NDVI has been dropping this is for every pixel and fit models  you can also very cool you can save the R square for every pixel so this is R square of the trend  analysis and now we can mask out only show only the pixels where the R square is higher than 0.25  so it means there's a not only there's a trend we fitted but this trend is really let's say  significant and now you see there's many pixels drop out and then this one is if you take the  probability for beta to be significant so less than zero so we can compute all these  images actually Landro computes usually from this trend analysis like 16 images can be different  components of coefficients of I don't know some if you want to this periodicity you can compute  these images and visualize them so yeah and then we can zoom in we can zoom in somewhere in France  I think Landro found two spots one spot is highly reddish and the other spots highly green  I don't know where these areas are now but you can see when you zoom in on the NDVI data it is clearly  this greenish there's a clearly increase in NDVI and here clearly there's a decrease and I think this  this area here it's close to some place I'm not sure so you we will have to check but it could be  that there were forest plantations and then they cut them down or something so there was a clear cut  but in Brazil also you know there's a lot of conversion of land into agriculture if you zoom  in also you can clearly see that places where they convert forests to agriculture that you lose NDVI  and so we analyze this data and this data is available and you can see that the cities as  they grow they cause NDVI to drop of course and so it matches lots of things matches what we expect  so sorry can I just start off why would India or China get greener?  Because they intensify in agriculture they put more fertilizer in the soil  and they had to produce more food and they get greener  the question is why did China and India why the greener  is that correct Leandro right? It's more intensive  production so they have also they rotate through year you know you can do two cropping season or  two harvest or three harvest season so they intensify in production  this is what we did for the NDVI then we said okay let's play let's do the same thing for the  temperatures so exactly the same you remove the seasonality you fill in the gaps you remove  seasonality and we did the same thing and people went nuts about this and again you can see we can  do it for the daytime temperature nighttime temperature and you can see that the China India  is getting cooler because they may get more vegetation so daytime temperature get cooler  but the nighttime temperature are more neutral so they're not they're not changing so much  but in general you can see here what you see as a trend blue is a negative red is positive  what is your first impression?  There is a global warming obviously there's more red than blue  and where does the global warming happens the most?  It looks like it's here Russia and Alaska and all the places where there's for example if you  look here this reddish areas this is a change in temperature because they cut the forest and they  put the cropland and you see there's all this basically kind of a front line of the conversion  of the land all this front line you see reddish so there's a really change a change in temperature  it's also interesting to see difference between a big difference between daytime and nighttime  temperature and this is a zoom in on Montpellier I have this map I will show you that tomorrow you  can access it and you can see the red is red and blue again and you can see there's a in general most  of France is the same temperature but somehow this place here for example where they had these  plantations of eucalyptus or something there's a wetlands here I think this is the  French Basque country and looks like there's an increase in temperature for some reason also the  mountains are warmer Pyrenees seem to be warmer and all this area here seems to be getting warmer  but gently not not a huge increase okay tiger mosquito so tomorrow I will show you now this was  a environmental data time series so tomorrow I will show you the whole case study we get the G-BIF  data on mosquitoes I think we start with like 50 000 records of these mosquitoes from G-BIF then  we subset to Europe then we finish with 28 000 I think and so we have 28 000 points it's a significant  data set for each point we know the the date the year and we know the individual counts so they  even have the counts of mosquitoes and that's the stuff I put also in this dashboard that  Francesca was showing and so what we do then we analyze this mosquito spread using a space  time machine learning so we do this model B we produce the time series of images of occurrences  of mosquito and once we produce these images of occurrences then we can visualize them and you see  this is now different years going from 2000 to 2021 and you can see that it is spreading the  mosquito is slowly spreading towards the inside I'm just showing Spain but so we get that red  color is of course high probability of occurrence and a yellowish color is slow and so you can see  it spreading yes so I can show you that's the thing I will I will talk to more about I can show  us today so this is all in this this tutorial and so we well it's tricky we had to generate this  pseudo absences about 10 percent pseudo absences and once we create pseudo absences then we can  this is the pseudo absences we generate them by using the max like package and it is highly the  mosquito is climatologically it follows specific areas very clearly so you can generate this  pseudo absences these are the pseudo absences and then when we do the modeling when we do the  modeling then we take about 45 covariates so we have the temperature rainfall nightlight images  human impact yeah lots of lots of images snow etc and then when we fit the model the model is  significant and what comes up is the most significant covariates actually night lights  night lights time series of night lights and then travel time to cities and ports that comes on the  top of the list and then once we make that then we can do predictions and we produce  predictions based on sorry question so the occurrence only data that's what the biologist  like the most it's basically this is the occurrence only data right basically it's fine except you  cannot do any machine learning with it I mean I as least I don't know I don't know which which  what would you use so because the anything in machine learning it's like either classification  regression survival problems if you look at for example classification problems you cannot have  only one state you have to have at least two states to start or multiple states so you have  like a zero and one state you know where it comes and where it doesn't come so with this thing you  only know where it comes but you don't know where it doesn't come so the machine learning cannot  train cannot be trained so we need to insert pseudo absences and the pseudo absences there's  lots of methods in the literature and most of them they they suggest to be very conservative so  only insert for example 10 so small amount of points and to locate it very conservatively  in places that you know the mosquito doesn't come one example is I don't know top of the alps  right minus 20 degrees I don't know then you know places like Iceland I don't know so you say  these are the pseudo absences and then model can at least know where it is absolutely not coming  and then once you make a map you get you get this you get these predictions and then it's fine  then you can because you can introduce the bias you can introduce if you the pseudo absences  are you know they're difficult to validate and we do them only to kind of bridge that gap so we  can do machine learning so and most of literature suggests to do to be conservative and limit to  like say 10 10 percent maybe if you do 20 percent I mean that's not a strict number but if you look  at the the literature so here's the this is the key reference if you look at this literature you  can read about it but number one they in this paper they they prove that if you do good quality  pseudo absences that it helps with modeling that's number one but they also say you should limit to  limit the pseudo absence only to something which is you're highly certain right because imagine if  I put a pseudo absence and let's say the mosquito does appear then I confuse my model and I introduce  a bias so but these are the pseudo do you have experience somebody has experienced  pseudo absences yes yes okay you can do hundred you could do hundred pseudo absences  it for machine learning would be enough I hear I do about 10 percent and it's a it's a really no  problem I get my you know I can see with the models when we do modeling for these mosquitoes  we got really distinct models and a good match you know we have a probability space we have 0.1  0.1 errors plus minus so that's a relatively good match you know you cannot put it in our square  because it's a binomial variable but when you when you look at the maps you can see it's really  distinct patterns and it's a it matches the the point data very well and we have the covariates you  know the the covariates we get also very clearly night light images the then the travel time to  cities you see night light images travel time to cities and so it kind of matches what we will expect  yes I have 150 I have 150 covariates I have 150 but it's computational to do selection so we went  and we went to just put 45 just for the exercise but there's a lot of covariates and covariates are  prepared as cloud authors you this I will show you you can connect you can also do your own analysis  for them you don't have to download them so I will show you they are available as a geospatial  database about 25 gigabytes of data we can put more if you need and they cover a time series from  so from 2000 to 2021 but we also have from 1985 we have the data so if somebody wants to go back  in the time it's also possible yes this one yes I think the ability of the satellite is by your  that could have affected so the about the NDVI analysis Landro was there any problems with  the modus in a sense that the images maybe make a bias so maybe the NDVI it's overestimated  because of modus sensors it's the same sensor now yes yeah modus it's a high quality product so  it's the same sensor running since 2000 so and it's like there are several reprocessing steps to  like remove the effect of the atmosphere effect so this NDVI was based in what we call like a  surface reflectance so it of course it tries to simulate how would be the response of the land  and conceiving like a sensor close to the surface so I don't see like bias in this  product but that's a very nice question because this product is the collection sticks so it's  running for for several years now and there is a paper that actually explained that there was a  significant difference between the collection five and the collection six so and they fix it in  the collection six so you couldn't see this kind of pattern in the collection five so because  every time that you have a new collection of the modus product they reprocess the whole archive  so and they are doing it for quite some years now so but in the collection six as far as I know it's  a high quality data so the answer to that question then to Frank is that most likely these NDVI trends  are correct and they do match the literature also they match also LAMSAT data NDVI etc  so I paper that you mentioned they actually analyze about three satellites I think they also  consider in spots and the probe of the so when all the three satellites are presented  a greener trend for the work and then there's a question about the was there any ground  routing so yes for NDVI it's there is for example extreme events they are documented for example fires  clear cuts of forest they're documented there let's say you don't have to go on the field you know  you can find this fire fire data sets and clear cuts and then you can compare and you can see that  you have a negative NDVI right the way you have a fire and or like a clear cut you will have a negative  negative beta in NDVI so yes there is ground routing not it sends that you go on the ground but you  could get for example high much high resolution images up to today you can get up to 30 centimeters  satellite images did you know that 30 centimeters and then with that you could see that there is  a really somebody cut the forest but there was a forest fire or landslide or flood and then the NDVI drops  please send us the link to the covariate sources no problem the covariate sources there is a CSV file  there's the CSV file I just have to open it in the github  so under inputs  and then CSV somewhere not here mood layers one kilometer all the layers are perfectly stacked  they're all 100 percent gap field and they're all perfectly stacked and they're all cloud  opt-in geotips and you can just grab them from here you can open them and put them and you can  see the source ID you can see what what are they what do they mean and you can read usually they're  all published in the publications at the moment there is about 500 almost 450 layers 450 but as I  said we have much more it's just I didn't want to overload you I mean we could put one and a half  thousand layers probably easily or two thousand layers and then these are all cloud opt-in geotips  running on S3 so you don't have to download it you can just make in a code connection  and then you can do extract crop overlay extract so you can do you can manipulate and it will only  pick up the it's like SQL query it will only pick up the data it needs it's a proper data cube  so okay this was my little contributions to time series analysis we look forward to showing you  how to do this in an hour and I have also I put actually two ways you can do this trend maps  one way is to use a package that will do it so there is the trend roster function in the green  green brown package and so is anybody using that package maybe it's for the for the images time  time series analysis image you can detect break lines you can set up the seasonality effects and  you can say that if you know for example if there's like 10 break lines you know you can  you can specify them so that's the package sorry break line is let's say when you have a  graph change in values in the time series data so that will be a clear cut of a forest or like a fire  or flood yes what they say break point yeah so that's the green green brown package but it  doesn't run in parallel and you can run it it's also a bit trouble to install it it's not on  it's not on cron anymore I think so it's a bit more trouble to to make it so what I did here  this is horrible code but look at this this is running linear modeling in parallel for every  pixel so you make a data cube and you have time series like let's say 30 years I don't know  and then you can fit any model you like for every time series and it will run in parallel  so I analyze here like 1 million pixels something like that maybe with less analyze it in 20 seconds  with with 32 threads so in 20 seconds poof done if I had more trends it's two seconds done  and this is the code to do parallelization but it's a bit abstract it's you know and  you know and when parallelization there's yeah there's steps you have to it's a bit abstract so  but the the important thing it's in our it's possible to crunch really large datasets  and you get this code is fully scalable I can apply it to any type of problem I have to do  time series analysis of you know data stacks so I could put also different models maybe something  that team was showing these models with this seasonality conversion I don't know so I could put  a lot of stuff to model  okay so yeah that's for me
MOOD_Summer_School_5_Timoth__e_Dub_Tom_Hengl_0_5531.png MOOD_Summer_School_5_Timoth__e_Dub_Tom_Hengl_5532_14735.png MOOD_Summer_School_5_Timoth__e_Dub_Tom_Hengl_14736_14783.png MOOD_Summer_School_5_Timoth__e_Dub_Tom_Hengl_14784_15671.png MOOD_Summer_School_5_Timoth__e_Dub_Tom_Hengl_15672_17219.png MOOD_Summer_School_5_Timoth__e_Dub_Tom_Hengl_17220_19091.png MOOD_Summer_School_5_Timoth__e_Dub_Tom_Hengl_19092_22943.png MOOD_Summer_School_5_Timoth__e_Dub_Tom_Hengl_22944_23831.png MOOD_Summer_School_5_Timoth__e_Dub_Tom_Hengl_23832_24911.png MOOD_Summer_School_5_Timoth__e_Dub_Tom_Hengl_24912_25739.png MOOD_Summer_School_5_Timoth__e_Dub_Tom_Hengl_25740_27083.png MOOD_Summer_School_5_Timoth__e_Dub_Tom_Hengl_27084_30131.png MOOD_Summer_School_5_Timoth__e_Dub_Tom_Hengl_30132_30443.png MOOD_Summer_School_5_Timoth__e_Dub_Tom_Hengl_30444_30491.png MOOD_Summer_School_5_Timoth__e_Dub_Tom_Hengl_30492_30731.png MOOD_Summer_School_5_Timoth__e_Dub_Tom_Hengl_30732_30791.png MOOD_Summer_School_5_Timoth__e_Dub_Tom_Hengl_30792_30875.png MOOD_Summer_School_5_Timoth__e_Dub_Tom_Hengl_30876_30971.png MOOD_Summer_School_5_Timoth__e_Dub_Tom_Hengl_30972_31199.png MOOD_Summer_School_5_Timoth__e_Dub_Tom_Hengl_31200_32039.png MOOD_Summer_School_5_Timoth__e_Dub_Tom_Hengl_32040_32291.png MOOD_Summer_School_5_Timoth__e_Dub_Tom_Hengl_32292_33191.png MOOD_Summer_School_5_Timoth__e_Dub_Tom_Hengl_33192_33515.png MOOD_Summer_School_5_Timoth__e_Dub_Tom_Hengl_33516_36479.png MOOD_Summer_School_5_Timoth__e_Dub_Tom_Hengl_36480_37055.png MOOD_Summer_School_5_Timoth__e_Dub_Tom_Hengl_37056_37103.png MOOD_Summer_School_5_Timoth__e_Dub_Tom_Hengl_37104_37955.png MOOD_Summer_School_5_Timoth__e_Dub_Tom_Hengl_37956_38159.png MOOD_Summer_School_5_Timoth__e_Dub_Tom_Hengl_38160_39359.png MOOD_Summer_School_5_Timoth__e_Dub_Tom_Hengl_39360_40727.png MOOD_Summer_School_5_Timoth__e_Dub_Tom_Hengl_40728_42323.png MOOD_Summer_School_5_Timoth__e_Dub_Tom_Hengl_42324_43103.png MOOD_Summer_School_5_Timoth__e_Dub_Tom_Hengl_43104_45731.png MOOD_Summer_School_5_Timoth__e_Dub_Tom_Hengl_45732_46211.png MOOD_Summer_School_5_Timoth__e_Dub_Tom_Hengl_46212_52415.png MOOD_Summer_School_5_Timoth__e_Dub_Tom_Hengl_52416_54659.png MOOD_Summer_School_5_Timoth__e_Dub_Tom_Hengl_54660_55187.png MOOD_Summer_School_5_Timoth__e_Dub_Tom_Hengl_55188_68963.png MOOD_Summer_School_5_Timoth__e_Dub_Tom_Hengl_68964_69035.png MOOD_Summer_School_5_Timoth__e_Dub_Tom_Hengl_69036_77231.png MOOD_Summer_School_5_Timoth__e_Dub_Tom_Hengl_77232_77495.png MOOD_Summer_School_5_Timoth__e_Dub_Tom_Hengl_77496_77795.png MOOD_Summer_School_5_Timoth__e_Dub_Tom_Hengl_77796_83195.png
What I would like us to achieve within BISARW is to understand the relevance of time series analysis of disease data to learn how to identify and take into account periodicity or cycles of seasonality you can use as many different names as you wish, how to identify your trend in your disease data and how to investigate a relationship between two time series we will be very very short on that I think it will be just one slide but that's the simplest stuff once you understood the rest. A disclaimer is that a whole week is not enough so you'll just have one hour and I hope you will get some nice information and enough knowledge to try and get started doing some time series analysis with data that would make sense after this hour and a half I've inserted a few bits and pieces of our code here and there that you can copy paste around but I guess you all know are much better than I do and I've stolen that most of those slides to some colleagues from ECDC so I really acknowledge them here so I had to be a pain in the ass with Willy through that slide and now we get to the real stuff so time series data is data points collected over an interval of time like so several data points at different time points the successive observations are usually not independent if it's 35 degrees today it is very likely that yesterday it was not minus five so those observations are not independent they kind of follow each other each others so what do we use it for in EP we use it for surveillance description of data and reporting to generate some hypothesis regarding seasonality or transmission cycles transmission periods and once again information for action we use it to detect outbreaks and respond early we use it to allocate resources healthcare facilities and stuff we can also use it for forecasting and for evaluating interventions we will not be touching forecasting today so a few examples here the evaluation of an intervention and in 1998 they screen poultry stocks for some salmonella types at regular intervals and if positives they were they would slaughter the whole flock he'd treat egg clean disinfection and of the breeding area that's an intervention which according to our time series analysis was clearly effective on the serotypes that were of salmonella that were targeted as you can see in the stuff below you can also use it for outbreak detection and you're going to use some models some forecasting models with some prediction intervals that will tell you whether you have more cases that you should observe or that you should expect so you can have some threshold for early detection and response and there is an example of an art package which I have never used but it this one was recommended to me you also have other activities like heat wave preparedness that can be conducted with time series but once again it all comes back to the quality of the data that's the surveillance and epidemiologist in me saying that you need to assess whether your data is good if it's representative of your surveillance system the idea like will access to diagnostics will be similar in all the area is the reporting as timely across the whole country I don't know if you've ever used a register like infectious disease register data but you usually have several options for a date you've got the date of sampling you've got the date of notification you've got the date of suspicion well if your register has that stuff I'm interested but usually you can have the date of first consultation for seeking of care you can you can have that in some registers so you need to think about the date you wish to use and this is something to keep in mind especially if some in some areas the reporting can be delayed and there's also something a bit weird and funny is that we're attracted to round numbers in the late 90s in the US they looked at data onset from diseases notified and you have twice more on the first day of each month that's a reporting issue and that's something you will have to think of when working on your analysis with time series data if you're using surveillance data and if the quality is not that great so garbage in garbage out the quality of the data thank the epidemiologist if the data is good thank the data managers who run the surveillance system do not call the surveillance system a cow with makeup some colleagues will let you know about that so how do you want the data that you will be using to be you want it to be aggregated you will need to think about the time units you want the time unit that is of relevance for your work so I like to use month because I usually work on small stuff like tickborn and cephalitis cases in Finland but for influenza transmission or for something like covid weekly basis can be quite interesting you will need to think about your areas also especially if you want to do some panel data analysis and if you're considering using covariates such as weather if you have a very large area you will totally dilute the effect of that far of that parameter so you always have to take everything into consideration there are a few art packages of interest that I've listed here you can play around it during the crash course and I will stay for the crash course maybe depending on my luggage gets here so the time series has four components three and a half you have the trend which is whether an event increase or decreases the frequency of an event decreases or increases over time and then you have what I call what we can call periodicity is whether you're the occurrence of the event follows some cycles cycles are supposed to be defined as above a year and then you have the seasonal component which is what happens during a short period less than a year and then you have the random component this stuff when you will work on your model will most likely be your residuals daily so if we're thinking disease data you will have it would be a c if you're into disease data we're not going to look at the variations within the date which is why I've never prepared for that question so it's it would be a something like a periodicity but I would not call it a cycle for me they're both periods but in AP they like to call seasonal something that's below a year and cycles something that's over several years so if you decompose your time series data it's built up of three things the trend of a time which here is a bit of the influx the periodicity because we're looking I forgot the scale here shame on me because we're dealing with something that's yet that appears to be like a 52 weeks cycle and then you will have your residuals your random component so what's the trend that's a bit too simple for you maybe so it's the long-term direction of the time series you have a trend if there is a long-term increase or decrease your data can also have no trend you can have stuff that's super stable and a trend is not necessarily linear so how will you identify your trend super easy you plot your data how do you analyze it plenty of options we look so it can be a bit complicated if you look for example at weekly cases of legionellos here legionellos is here it can be a bit complicated to see if there is a trend in that situation because it's it's rather flat but if you have a quick and dirty linear regression linear regression you see that you have an upward increase over time and well this was too simple we can skip and that was the same again so still as I was saying when you don't have your linear stuff it's a bit complicated to have a direct idea of what your trend would be so step one you plot your data that's fairly easy I think you all know that already step two you use you can use different techniques to analyze the trend and the easiest would be to smooth to smooth it up using a moving average what's a moving average will be the next question you in EP we use centered moving average so for example here we have monthly cases of the disease you will use here it's a five weeks moving average that you're using so you're centering of the week of interest and take the two weeks before and two weeks after and you do the average of that this will give you something that doesn't have those very very that's those way too many variations this gives you an idea of the trend over time so it also means if you want to investigate a disease over the year 2020 to 2022 when you will ask for the data to your friendly data manager you ask for two months before the beginning of your period of interest after because if you come back to him two weeks later when you want that and if you've called his register a car with makeup he will not be happy so that's a fairly simple example moving average it will not be a problem for you and the wider window you choose the flatter and stable your curve will appear this is just for you to see what's happening for your plotting and you can see that with a 51 weeks window over a period of more than 10 years you see a clear very slight increase over time which might be very difficult to to see with a five weeks window or with that 25 week window are you still following okay good now simpler solution if just like me you can't see anything when you look at those kind of graphs go for a very simple linear regression where you will have well you will just put your time variable so year week if you're looking at something after weekly data or month into your model and then you will have a coefficient for that and this you don't need I think and this yeah so and you will get your coefficient that gives you the coefficient of increase of cases over time I have a personal preference for using negative binomial regression instead of linear because it's less cumbersome regarding assumptions that will just touch it up a bit later then you have the difference between your observed and your predicted your residuals and we'll discuss it a bit later so you're have a few assumptions when using a linear regression that setting and but also when assessing a trend you want the association between the two variables to be linear except that if you have an intervention that occurs at one point you might want to add this into the month into your model and that would be another topic you want the your errors to be independence you want your horrible horrible that you have to respect normality so for each value of x the value of y are normally distributed and that's where I'm a little bit out of my depth being more an epidemiologist who will use and look at the stuff and be like it doesn't look like it's supposed to look and just go ahead and do something else and then you want to have an equal variance over time so here is a simple linear regression where they looked at assessing the trends and your coefficient will be what you're interested in I'm sorry this is a stator output I didn't have time to redo them with r and that coefficient will give you the slope of your trend over time and here you will not look at p values so this is how your simple linear regression actually looks like for that data now we have to talk about the residuals you know probably all more about modeling than I do but your residuals is the difference between what you've observed what you've seen in real life and what your model estimates and you want so much stuff from those residuals you want them to follow normal distribution you want their variance to be around zero and for example this I'm not happy with it so you need to find other options other models that have less assumptions than issue or you can log the stuff to reduce the spread of the residuals and you end up having something that looks way much nicer and that's the type of residuals graph I'm happy with but the problem with that logging is that the interpretation of coefficients will change and it will get a bit more complicated so with each unit increase in the dependent variable you will have a multiplicative percentage change in independent variable seriously log transformation I'd rather go for negative polynomial always so some key ideas in your trend analysis you plot your series and you're moving at you look at the structure you describe it you decide whether a log transformation is needed and you decide not to do it you fit your regression model you interpret you check your residual and make sure you interpret your coefficients correctly the numbers need to make sense in the real world 0.0001 percent increase over time per month or year is not that interesting except if you're working on a million years data okay so next component the cycles and the seasons here you can see in that data that you have some cycles over a period of three to four years this is exactly what you want to look at and take into account when modeling time series data and this is how a model would look like and how we observe with your flag the seasonal variation is within a year and they tend to repeat themselves each year that's Campylobacter for example and you can clearly see it and that's a brand new one that we had a quick and dirty look this week it's chlamydia infections notified in Finland so you can see that COVID restrictions didn't do much to the spread of chlamydia over the past two years if you know anybody who's into zero COVID dogma you can tell them that if well yeah just let them have a look at that and then you will have your random variation which I wanted to talk about a little bit later but so your classic time series is the trend the cycle if there is a cycle if you have long enough period to see that because if you only have two years and if there are cycles that are way larger you won't get them the seasonal periodicity and then the random stuff it's every time series data plot can be decomposed as follows and by adding all that stuff the trend the cycle the noise and everything you might end up having something not too far from the real life and this is an example of something we did for tickborn encephalitis in Finland over 11 years and as you can see we had that trend over time not linear we had those trend plus a 12 month cycles that you can see here and as you can see we are not too bad at predicting the incidence of TBE except we don't really know what happened afterwards in the two years that followed we had way more of an increase and and now my model sucks so how do you assess the seasonality other than by plotting and that's where I've copy pasted some R code the spectral analysis which is a very fancy word to say periodogram is what you want to do first and there is that very no yes so if you have a data set with cases per week weekly cases or monthly cases and but keep in mind that this doesn't work with panel data you need whole aggregated so one observation per period you have that very easy formulas from it's actually from base R that will allow you to plot a periodogram and you will have to decide on a period so here for example we decided that we wanted the scale of the periodogram to be 52 weeks and if you look at that I have that periodogram with two peaks plus a third one those two peaks are the main period which is actually because we decided of the 52 week scale 52 weeks divided by one so a 52 week period and a 26 weeks 52 weeks divided by two another 26 week period that you get in there and that's when you start with the fun stuff because every curve that's that is a cycle can be decomposed in sine and cosine so when you want to add a periodicity or a seasonality to your model then you will just generate a sin a sinus and a cosine of those periods key message you plot your series you detect and describe a pattern you play around with sine and cosine then you interpret your coefficients in terms of something that has to make sense those 0.0.0.0.1 increase per 10 years you're not really interested try to keep it simple and then check your residuals but then because of all that crap with residuals that always make me very unhappy you have another possibility you can consider other models that linear regression and I would say that I have a strong preference for negative binomial regression because you have less assumptions to bother with and more veteran statisticians would tell you that now that we have the computer power to run them quite easily on almost any laptop it's quite easy to go straight to that and it allows epidemiologists to pretend to know what they're doing which I really like so what I will talk about also time series but and now it's a space time so it's a time series but we work basically with the both space and time and there are basically two versions of analysis can do when you have both space and time you can do simultaneous analysis of trends so one of the option is this spar package in R you can do you can feed this space time densities and but this will only work with the current only data so no quantities and then the second option is to first harmonize everything in space and then do the time and so I'm going to show you well tomorrow I will show you the spar a bit but our main focus is on this number two so the version number two which is first harmonize the space then analyze time series and then you focus only on time and then what we do basically we apply type of modeling that you saw from team we applied for every pixel so it becomes very very computational okay super computational and also the what's important to discuss is this so there's the seasonal components there's the cyclical components I'm not going to talk about it for me it's all kind of this sinusoidal or whatever curves then there's also breakpoints I think you you don't have that maybe in the health data but in vegetation data there's a breakpoints like when there's a fire or natural hazard and then also there are the distributions they also change and so that's also important when you run the analysis and let me see this is from the Wikipedia by the way these things from Wikipedia if you if you open it in a let's say so if you open this in Wikipedia you go to this image it's as usual in Wikipedia if you look at the meta information this looks like some real data set right no it's not a real data set it is let's see more details it's been generated through our code it's a simulated data so it's been generated it's a simple it's a it's a dummy data and they they played a bit so they wanted to show so if you use the raw data the simple thread line it's almost worthless so it's like r square 0.0 0.08 right but if you do the filtering then the model is 0 r square is 0.97 so the same basically same data set but you see there's a huge difference how you implement modeling going from no correlation to to low r square to high r square so yeah you can take a look at that but that's a classical data set and this thing also T mentioned there are these components so this will be a original signal and you can extract the seasonality and then you can maybe detect the trend and then the remaining things is the random component and random component is noise you don't want to model never fit a model to noise please that's a physics that's not statistics but yes seasonality and trend that's something we're interested and so as I said the team also mentioned that I was I will still say this noise noisy component yeah be very careful and I only put here one so I cyclic hollow seasonality I put it together but I agree there's more components here so number two there's a ABC I don't know there's more versions this is the global temperature measures for the last 10 000 years and also you see this I think uncertainty the the blue line is uncertainty and so what you see is how how the scale that you look at you know like a annual temperature the scale that you look at actually the world's been warming up a bit I think since the glacier period 10 000 years ago there was a cold cold period here and then the the world's been warming up and it's been actually cooling down it's been cooling down but now we have industrial evolution and it's the red line and you see it's a one degree and it's completely off the scale it's very difficult to even visualize it it's a really hockey stick but so that depends also this time series analysis depends how far you look at okay now the space time as I said we do the we interest in space time analysis of trends and so now we have one data set my colleague Leandro here he's been running this analysis and we downloaded all of the modest data set actually it's almost 14 terabyte of data so we downloaded and we took this data set and we prepare like a data cube and then we run the time series analysis because we would like to see where is the the NDVI you know the normalize the difference vegetation index it's like a vegetation index it's a measure of primary productivity of landscape we would like to know where is the negative trend in NDVI where is the positive because that's the measure of land degradation let's say so we wanted to analyze that and we prepare like these data cubes as I said that's really can be really large especially if you do global we also prepare for Europe at 30 meter we have global at 250 meter and then we want to estimate this trend and then the first thing we realized that there's many missing values for example there's a lot of lot of missing values then also there's artifacts there's some outliers and and then okay we have to do some gap filling and so after we remove the outliers and we do gap filling then we can do trend analysis and as a result of trend analysis this is just fitting a linear model to data we can get the map of the slope this is the alpha the intercept so that's the beginning temperature the beginning NDVI at 2000 and then the slope it's a positive or negative negative means decrease of NDVI and decrease and positive red color here it means increase but as I said in between we have to we have to derive all this time series again and we have to actually convert all the layers first to make to fill in the gaps after we fill in the gaps then we remove the trend or the seasonality component because the NDVI changes you know winter summer for every pixel winter summer is a bit different right in the world and then once we remove this component then we can calculate per pixel and imagine this is a huge image it's like 150 000 by I don't know 100 000 pixels they're huge images and we take the time series of pixels and we run models through time for every pixel and we have to run it in parallel and it will take like a one day of computing with full capacity right Landro this NDVI will be like a one day and then so this is one this is one row data this is a big monthly image for for January and you see there's a lot of missing pixels if we will like to compute any trends on this data it will not be possible because we have too many missing pixels so we have to do this gap filling and then after gap filling trend deep trending and then the trend analysis so this is the one image this is our computing so this was with the 400 threads we have not 1000 threads so we can speed it up a bit but this is you see running on on servers processing in parallel every pixel time series running in parallel and this is the NDVI after gap filling you can see how the world our world breathes basically right so this is the vegetation changing Landro made this realization it's really mind-blowing right many people interested in that and it's nice to see it after you remove all the gaps you know and everything so it's nice to see it but you see there's a seasonality effect so if we will fit a trend line it will be difficult like in this first example to show this Wikipedia example it will be difficult to fit a trend line so we do this is one point with the NDVI this is the original NDVI this is the seasonality and then this is after removal after removal you see we have just the trend and these are the residuals of the of the model and so here we estimate that there's an increase in NDVI for this pixel in the last last 20 years there's an increase in NDVI so it could be that they started planting reforesting I don't know and this is the same image of the world the same the same months but after you use remove the seasonality do you see a difference I mean if you watch carefully you can see that there are some differences and these could be differences for example due to El Nino La Nina effects you know it could be also differences because of land degradation so there's a there's a multiple not things but this is now the seasonality is taken out so you can see now also a feature which is basically something that the team explained only temporarily now we looked at it in the space we look at the space what happens with vegetation last 20 years taking out the seasonality and then we can compare of course the beginning and the end NDVI that's also one way to detect but this wouldn't wouldn't work because by accident you could have a difference between begin and end by by accident could be small difference but through the time it's actually bigger and then this is the result ordinary least square fitted for every pixel red color red color means it's a decrease in NDVI a green color it's an increase and so what happens what is the conclusion what do you see immediately the world is getting greener and it's been published in a big journals and and the world is getting greener because of actually two countries China and India exactly so you see there's we're getting greener Europe is kind of let's say neutral but there is Eastern Europe is also looks like it's getting greener but there are places where there's reddish pinkish and you see there is some spots where obviously the NDVI has been dropping this is for every pixel and fit models you can also very cool you can save the R square for every pixel so this is R square of the trend analysis and now we can mask out only show only the pixels where the R square is higher than 0.25 so it means there's a not only there's a trend we fitted but this trend is really let's say significant and now you see there's many pixels drop out and then this one is if you take the probability for beta to be significant so less than zero so we can compute all these images actually Landro computes usually from this trend analysis like 16 images can be different components of coefficients of I don't know some if you want to this periodicity you can compute these images and visualize them so yeah and then we can zoom in we can zoom in somewhere in France I think Landro found two spots one spot is highly reddish and the other spots highly green I don't know where these areas are now but you can see when you zoom in on the NDVI data it is clearly this greenish there's a clearly increase in NDVI and here clearly there's a decrease and I think this this area here it's close to some place I'm not sure so you we will have to check but it could be that there were forest plantations and then they cut them down or something so there was a clear cut but in Brazil also you know there's a lot of conversion of land into agriculture if you zoom in also you can clearly see that places where they convert forests to agriculture that you lose NDVI and so we analyze this data and this data is available and you can see that the cities as they grow they cause NDVI to drop of course and so it matches lots of things matches what we expect so sorry can I just start off why would India or China get greener? Because they intensify in agriculture they put more fertilizer in the soil and they had to produce more food and they get greener the question is why did China and India why the greener is that correct Leandro right? It's more intensive production so they have also they rotate through year you know you can do two cropping season or two harvest or three harvest season so they intensify in production this is what we did for the NDVI then we said okay let's play let's do the same thing for the temperatures so exactly the same you remove the seasonality you fill in the gaps you remove seasonality and we did the same thing and people went nuts about this and again you can see we can do it for the daytime temperature nighttime temperature and you can see that the China India is getting cooler because they may get more vegetation so daytime temperature get cooler but the nighttime temperature are more neutral so they're not they're not changing so much but in general you can see here what you see as a trend blue is a negative red is positive what is your first impression? There is a global warming obviously there's more red than blue and where does the global warming happens the most? It looks like it's here Russia and Alaska and all the places where there's for example if you look here this reddish areas this is a change in temperature because they cut the forest and they put the cropland and you see there's all this basically kind of a front line of the conversion of the land all this front line you see reddish so there's a really change a change in temperature it's also interesting to see difference between a big difference between daytime and nighttime temperature and this is a zoom in on Montpellier I have this map I will show you that tomorrow you can access it and you can see the red is red and blue again and you can see there's a in general most of France is the same temperature but somehow this place here for example where they had these plantations of eucalyptus or something there's a wetlands here I think this is the French Basque country and looks like there's an increase in temperature for some reason also the mountains are warmer Pyrenees seem to be warmer and all this area here seems to be getting warmer but gently not not a huge increase okay tiger mosquito so tomorrow I will show you now this was a environmental data time series so tomorrow I will show you the whole case study we get the G-BIF data on mosquitoes I think we start with like 50 000 records of these mosquitoes from G-BIF then we subset to Europe then we finish with 28 000 I think and so we have 28 000 points it's a significant data set for each point we know the the date the year and we know the individual counts so they even have the counts of mosquitoes and that's the stuff I put also in this dashboard that Francesca was showing and so what we do then we analyze this mosquito spread using a space time machine learning so we do this model B we produce the time series of images of occurrences of mosquito and once we produce these images of occurrences then we can visualize them and you see this is now different years going from 2000 to 2021 and you can see that it is spreading the mosquito is slowly spreading towards the inside I'm just showing Spain but so we get that red color is of course high probability of occurrence and a yellowish color is slow and so you can see it spreading yes so I can show you that's the thing I will I will talk to more about I can show us today so this is all in this this tutorial and so we well it's tricky we had to generate this pseudo absences about 10 percent pseudo absences and once we create pseudo absences then we can this is the pseudo absences we generate them by using the max like package and it is highly the mosquito is climatologically it follows specific areas very clearly so you can generate this pseudo absences these are the pseudo absences and then when we do the modeling when we do the modeling then we take about 45 covariates so we have the temperature rainfall nightlight images human impact yeah lots of lots of images snow etc and then when we fit the model the model is significant and what comes up is the most significant covariates actually night lights night lights time series of night lights and then travel time to cities and ports that comes on the top of the list and then once we make that then we can do predictions and we produce predictions based on sorry question so the occurrence only data that's what the biologist like the most it's basically this is the occurrence only data right basically it's fine except you cannot do any machine learning with it I mean I as least I don't know I don't know which which what would you use so because the anything in machine learning it's like either classification regression survival problems if you look at for example classification problems you cannot have only one state you have to have at least two states to start or multiple states so you have like a zero and one state you know where it comes and where it doesn't come so with this thing you only know where it comes but you don't know where it doesn't come so the machine learning cannot train cannot be trained so we need to insert pseudo absences and the pseudo absences there's lots of methods in the literature and most of them they they suggest to be very conservative so only insert for example 10 so small amount of points and to locate it very conservatively in places that you know the mosquito doesn't come one example is I don't know top of the alps right minus 20 degrees I don't know then you know places like Iceland I don't know so you say these are the pseudo absences and then model can at least know where it is absolutely not coming and then once you make a map you get you get this you get these predictions and then it's fine then you can because you can introduce the bias you can introduce if you the pseudo absences are you know they're difficult to validate and we do them only to kind of bridge that gap so we can do machine learning so and most of literature suggests to do to be conservative and limit to like say 10 10 percent maybe if you do 20 percent I mean that's not a strict number but if you look at the the literature so here's the this is the key reference if you look at this literature you can read about it but number one they in this paper they they prove that if you do good quality pseudo absences that it helps with modeling that's number one but they also say you should limit to limit the pseudo absence only to something which is you're highly certain right because imagine if I put a pseudo absence and let's say the mosquito does appear then I confuse my model and I introduce a bias so but these are the pseudo do you have experience somebody has experienced pseudo absences yes yes okay you can do hundred you could do hundred pseudo absences it for machine learning would be enough I hear I do about 10 percent and it's a it's a really no problem I get my you know I can see with the models when we do modeling for these mosquitoes we got really distinct models and a good match you know we have a probability space we have 0.1 0.1 errors plus minus so that's a relatively good match you know you cannot put it in our square because it's a binomial variable but when you when you look at the maps you can see it's really distinct patterns and it's a it matches the the point data very well and we have the covariates you know the the covariates we get also very clearly night light images the then the travel time to cities you see night light images travel time to cities and so it kind of matches what we will expect yes I have 150 I have 150 covariates I have 150 but it's computational to do selection so we went and we went to just put 45 just for the exercise but there's a lot of covariates and covariates are prepared as cloud authors you this I will show you you can connect you can also do your own analysis for them you don't have to download them so I will show you they are available as a geospatial database about 25 gigabytes of data we can put more if you need and they cover a time series from so from 2000 to 2021 but we also have from 1985 we have the data so if somebody wants to go back in the time it's also possible yes this one yes I think the ability of the satellite is by your that could have affected so the about the NDVI analysis Landro was there any problems with the modus in a sense that the images maybe make a bias so maybe the NDVI it's overestimated because of modus sensors it's the same sensor now yes yeah modus it's a high quality product so it's the same sensor running since 2000 so and it's like there are several reprocessing steps to like remove the effect of the atmosphere effect so this NDVI was based in what we call like a surface reflectance so it of course it tries to simulate how would be the response of the land and conceiving like a sensor close to the surface so I don't see like bias in this product but that's a very nice question because this product is the collection sticks so it's running for for several years now and there is a paper that actually explained that there was a significant difference between the collection five and the collection six so and they fix it in the collection six so you couldn't see this kind of pattern in the collection five so because every time that you have a new collection of the modus product they reprocess the whole archive so and they are doing it for quite some years now so but in the collection six as far as I know it's a high quality data so the answer to that question then to Frank is that most likely these NDVI trends are correct and they do match the literature also they match also LAMSAT data NDVI etc so I paper that you mentioned they actually analyze about three satellites I think they also consider in spots and the probe of the so when all the three satellites are presented a greener trend for the work and then there's a question about the was there any ground routing so yes for NDVI it's there is for example extreme events they are documented for example fires clear cuts of forest they're documented there let's say you don't have to go on the field you know you can find this fire fire data sets and clear cuts and then you can compare and you can see that you have a negative NDVI right the way you have a fire and or like a clear cut you will have a negative negative beta in NDVI so yes there is ground routing not it sends that you go on the ground but you could get for example high much high resolution images up to today you can get up to 30 centimeters satellite images did you know that 30 centimeters and then with that you could see that there is a really somebody cut the forest but there was a forest fire or landslide or flood and then the NDVI drops please send us the link to the covariate sources no problem the covariate sources there is a CSV file there's the CSV file I just have to open it in the github so under inputs and then CSV somewhere not here mood layers one kilometer all the layers are perfectly stacked they're all 100 percent gap field and they're all perfectly stacked and they're all cloud opt-in geotips and you can just grab them from here you can open them and put them and you can see the source ID you can see what what are they what do they mean and you can read usually they're all published in the publications at the moment there is about 500 almost 450 layers 450 but as I said we have much more it's just I didn't want to overload you I mean we could put one and a half thousand layers probably easily or two thousand layers and then these are all cloud opt-in geotips running on S3 so you don't have to download it you can just make in a code connection and then you can do extract crop overlay extract so you can do you can manipulate and it will only pick up the it's like SQL query it will only pick up the data it needs it's a proper data cube so okay this was my little contributions to time series analysis we look forward to showing you how to do this in an hour and I have also I put actually two ways you can do this trend maps one way is to use a package that will do it so there is the trend roster function in the green green brown package and so is anybody using that package maybe it's for the for the images time time series analysis image you can detect break lines you can set up the seasonality effects and you can say that if you know for example if there's like 10 break lines you know you can you can specify them so that's the package sorry break line is let's say when you have a graph change in values in the time series data so that will be a clear cut of a forest or like a fire or flood yes what they say break point yeah so that's the green green brown package but it doesn't run in parallel and you can run it it's also a bit trouble to install it it's not on it's not on cron anymore I think so it's a bit more trouble to to make it so what I did here this is horrible code but look at this this is running linear modeling in parallel for every pixel so you make a data cube and you have time series like let's say 30 years I don't know and then you can fit any model you like for every time series and it will run in parallel so I analyze here like 1 million pixels something like that maybe with less analyze it in 20 seconds with with 32 threads so in 20 seconds poof done if I had more trends it's two seconds done and this is the code to do parallelization but it's a bit abstract it's you know and you know and when parallelization there's yeah there's steps you have to it's a bit abstract so but the the important thing it's in our it's possible to crunch really large datasets and you get this code is fully scalable I can apply it to any type of problem I have to do time series analysis of you know data stacks so I could put also different models maybe something that team was showing these models with this seasonality conversion I don't know so I could put a lot of stuff to model okay so yeah that's for me
analysis of disease data to learn how to identify and take into account periodicity or cycles of disease data and how to investigate a relationship between two time series we will be data if you're using surveillance data and if the Finland but for influenza transmission or for techniques to analyze the trend and the easiest with panel data you need whole aggregated so one simultaneous analysis of trends so one of the time series and then you focus only on time and this is the seasonality and then this is after