Spatial data exploration in Jupyter notebooks
Graser, Anita
Recent developments in Python data visualization libraries (particularly HoloViews & GeoViews) enable data analysts and scientists to quickly and intuitively create interactive data visualizations. In this talk, we dive into examples of visualizing open government data from public web services as well as large movement datasets.
 Welcome, my name is Anita Grazer. This is my first FOSDEM, so I'm really excited to be here to talk to you about spatial data exploration in Jupyter notebooks and the path that led me to Jupyter notebooks. I started working with Python and GIS over 10 years ago when Python support was just being introduced to QGIS and IG GIS. And for a long time, particularly QGIS and POST GIS have been my go-to tools for anything data analysis. I just went to my blog and dug out some POST from way back 2011 because it basically describes how I've been working for many, many years. I did data analysis in QGIS or in POST GIS and afterwards I could, or in between, I could visualize those results in QGIS, have a look at each step, do the results look okay, can I continue to the next one and so on, always verifying or testing the hypothesis of the analysis along the way. And this was a really important component and still is in all my workflows. So I even went so far and developed a couple of plugins which you might be aware of for visualizing particularly movement data in an animated way, so that was the time manager plugin which has now been superseded by some core functionality in QGIS. So basically it played into the same need of visually exploring data. Of course, besides desktop GIS and spatial databases, I've always been aware of other Python libraries which could be used to write standalone data analysis. So of course I knew about Shapely, I knew about Fiona, but I rarely used them. And the reason for that was that they felt kind of redundant for my use. A lot of functionality that they provided was already included in the POST GIS analysis functions or in the QGIS API. So there was no, often no need to add it and also it was quite tricky to install sometimes it, but the main issue is for me it lacked the visual component that is needed to do exploratory data analysis. So I would always have to go to some different tools afterwards to look at the results that were produced and this would just break my analysis workflow. This situation has completely changed now, particularly with the introduction of Geopandas and Jupyter notebooks. There is now a way to work directly in Python, but also keep this visual component. However, the default mudplot lib visualizations that are created with Geopandas are still rather limited. So you can look at them at their website, on the Geopandas website and you will see they are quite straightforward, you just call plot and it gives you a map, but it's static. And by default, there is also no background map to check the geographic context of your data. There are some examples in the gallery which show how to add background maps, but they depend on another library contextily and it's also just a static background map without any possibility of interaction. So that was the status quo I was at roughly two years ago and I started looking into, okay, what are the options for doing it interactively? How can I actually zoom into the map to see a bit more detail, for example? And the first library that I looked at was Folium. I'm sure you've heard of it. Folium is based on leaflet and it basically allows you to create interactive maps and put vector data on top. But I don't want to talk about Folium because that's not where I ended up with. There I really found my new home and what really works well for me is Holovis and that's what I want to demo to you today. And Holovis is not just one library, it's a whole family of libraries and particularly noteworthy of interest to us today are the GFUs library and the DataShader library. So they are directly in the name. GFUs gives you the capabilities to map spatial data and it's very handy and it has a lot of capabilities like built in background maps and interactive maps that we are going to explore now in this presentation. The first example that I want to show you is for all of you who maybe haven't worked with GeoPandas yet, just a quick look into how you can interact with spatial data and how simple it is to use. I'm currently working on a small lab setup which uses open government data which is provided by the city of Vienna. So you can have a live demo. If you click on the link here in binder, it will actually launch this hosted JupyterLab instance and I will show you exactly what this can do. I'm just importing a couple of basic libraries here. You can already see the JupyterLab contextual help updating depending on which module I select here. It's super useful help. But what I really wanted to get into is of course the spatial data analysis component. So there is a WFS service hosted by the city of Vienna which I'm just going to query. I'm going to download a GeoChasen file here, particularly of electric vehicle charging stations. So basically downloading it here, you can see the Chasen file was downloaded. And with GeoPandas read file, I can read this GeoChasen file and it gives me a GeoData frame that I can directly plot here in the notebook. And you see, I get an interactive map with popups which show me the location and the designation. So some name for this station and I can add whichever columns I like to be displayed here in this one line of code. So this is what I mean if I say it's really simple to make interactive maps in the notebook with this library. Of course GeoPandas also has all kinds of analysis tools or spatial analysis tools. Here I have just a quick example with buffers and I'm also taking care of coordinate transformations here. So here you can see all the stations buffered with a 100 meter buffer. And if you zoom in, you see the station location itself plus the buffer drawn around it. And of course I can do that with multiple different data sets. And the really cool thing is you can plot these data sets then side by side just with the plus operator here. And now side by side it's synced. I can zoom in. I can compare different neighborhoods and how they are served for example by electric vehicle charging stations or by city bike stations. And you can play this as much as you want. So I've imported another library for Geocoding with Nominatin which is GeoPi and edit the address of my office and now I can have a look around my office in a buffer of 1 kilometer. How many charging stations do I have? Looks like there's two of them. So this is one basic example which I invite you to visit. If you just go to OGDAT lab you can launch the binder here. Obviously this is a very small data set, it comes from this web service directly. But trust me if I say we are not limited to small data sets. And the actual larger example that I want to demo you today is about a couple million data points instead of just a couple dozen. So let me import the setup for this demo. Again this is mostly based on GeoPandas and also a lot on HoloViews and DataShader particularly because that's what we will need to plot the large data sets. As you can see here based on the code this is an example with AIS data which is Ship Movement data and I'm importing two days of this data. Eventually there's a lot of columns so there's also a lot of values which we want to get rid of because the speed is zero and that's rather boring for most analysis. So let's read in those two files. This is going to take a while but I wanted to show it to you how long it takes so that you can appreciate it later on when we work with the data how quickly it goes. You can also see here a couple of nice features of Pandas reading from text files. You can already specify which columns you want to keep, which you want to throw away. You can rename them so that they have a bit nicer column names to work with later on. I'm also specifying a temporal index right here. So I'm turning this dates, time, format strings into temporal objects and assigning them as the index. And even though it only shows the date here it's actually a date time so the time information is not getting lost here. All in all we have now loaded 12 million records and it's special temporal. There's an X and a Y column here. To be worth mentioning I have used a utility function of data shader which converts latitude and longitude to meters in Psojdomerkarta which then makes it easier to plot them on a base map. And that's exactly what we are going to do. We will want to plot them. First we can have a look at the basic extent. Obviously this is rather fast. It's just pandas accessing the column min and max values. But now let's plot this. Let's see how it actually looks like on a map. And we've now plotted 12 million points on a map and it's interactive. So we can zoom in. We can see all the points near Null Island. Probably some outliers we want to get rid of. But we can also see lots of points in other areas, our main areas of interest. We can crop this now. We can analyze it in all kinds of different ways. For example we might want to focus only on this small area, this bounding box. Again pretty fast and interactive. The default plots of this data shade are actually density maps. So what's usually called heat maps. If you compare this example I basically told him to not use a heat map because I used a color map from blue to blue. And I did an overwrite of the heat map to just get every individual point to display here. But if you don't provide a heat map it will give you this kind of visualization where you can actually see the density of the points and there you can very easily spot that it's much denser in this area of northern Europe than in all the other areas where there's also some data. Of course since we are in the Python world there's also really good functionality to do temporal analysis so we can have a look at how many data sets for example we have per time interval of 15 minutes or whatever you want to specify here. So in this movement data set you can see that on the 1st of January it's 1st of July it's super busy and on the 2nd there's much less traffic according to this data set. And having all these kinds of different analysis in the same notebook obviously has huge advantages for making it reproducible for documenting the whole data exploration process. Again you can have multiple plots side by side, you can have multiple maps side by side like I'm doing here where you can compare two different days. That'd be very interesting because it looks like the spatial extent of the data set has changed and on the 2nd we have this whole area here in the west which is basically empty for some reason. So it's really cool, it's fast, there is however a couple of downsides so there is no convenient way to visualize different categories so I can't say ships of ship type A should be red and ships of ship type B should be blue for example. Instead the only way that I found so far is to plot multiple scatter plots on top of each other or next to each other so I can of course filter the whole data frame by different categories and I can plot them side by side or like shown on the right or on top of each other as shown on the left but I have to plot each of those filtered data frames individually and I have to give it a color. And I cannot do just a one liner which says okay take this data frame and based on this column apply some chosen colors to each category. Another feature of data shader that I want to share with you is how you can also work with line features so until so far we have only plotted point features but obviously it also can plot line features and it has the same fast rendering which is achieved by the underlying implementation. And the example that I want to show you here is how to find segments that have gaps that represent gaps because there is a huge time difference between consecutive positions. And what you can see here in the next map once it's drawn is all those segments where there is a long jump between one location and the follow up location. And data shader obviously doesn't just work on maps it also works on all kinds of other plots so if you have a normal scatter plot for example the coordinate changes from one location to the other you can also plot those and irrespective of the coordinate system you now have another data shaded plot to explore. Something also related to this issue with not being able to plot categorized values is that you cannot visualize the attributes of line strings along the you cannot visualize the properties of line strings also categorized or graduated. And so these plots that I have included in the end of this notebook which are showing you the speed along a line string or along a trajectory those wouldn't be possible as far as I know with data shader right now so this uses the basic geo-fuse plotting instead and you can of course also zoom in here and this is vector data that is rendered here as opposed to the data shaded data which visualizations which are always rasterized in the background to allow for this quick rasterization. So sometimes I still have to go back and have to use the basic geo-fuse it's way more convenient if you want to have access to the attributes and visualize those as well. So if you're interested in more details on how to analyze movement data I invite you to visit MovingPanda's on GitHub and go to the example section and one of the examples that you can find here is a data set from a horse GPS caller that Lena Fischer from the University of Copenhagen took provided to me and if you go through the examples you will see a lot of these maps side by side with different backgrounds for example temporal analysis but what I really wanted to point out is also the capabilities of splitting the observations into different time frames and then all doing all these evaluations for example daily trajectory length or the daily covered area and you can see the existing or non-existing trends of these properties of movement for example and you can also have a look at to where the moving objects for example spent a certain amount of time which is called stop detection analysis and always the really neat part and what I want to emphasize is you just have a couple of lines of code and then there's the visualization of the intermediate results which should provide you really fast feedback and the possibility to dive deeper into data than you usually would if this whole workflow was more complicated to do so definitely ease of use I think makes for better data exploration because you just get a feeling that you can do more things and on this last note you getting the feeling of being able to do more things there's one feature that I want to highlight in the end and that's not just having interactive individual plots that you can zoom in that you can hover and see feature properties now you can also do linked plots and you can do filtering between different plots and I want to show this as an example between map plot so again one of the those scatter plots as we had before and histogram here is an example of a histogram of speed values so we want to link them together so that we can see for example which speeds are located where and this is really simple there is just one function that you have to use which is called link selection and it takes as input the two plots that we just provided created previously and what you can now do is you can filter here either on the spatial side and it will update the plot linked on the right side or you can filter here on the attribute side and it will filter also the geographic side and I think with this and some of the other tools like sliders and buttons that you can add the possibilities are basically limitless and on that note I want to encourage you to if you haven't yet I definitely give it a try I think it's a great addition to your toolbox of if you come from the more desktop GIS side of things and it certainly is also a great addition to your toolbox if you come from the classical data science and probably haven't done much yet with geographic data then I think this is the these are the libraries to get started with and if you are indeed working with movement data then feel free to reach out and share your experiences with moving pandas and with any of the notebooks I've shared with you today and I would be happy to exchange ideas and plans for future directions on these as well so with that back to you and thank you.
 Welcome, my name is Anita Grazer.  This is my first FOSDEM, so I'm really excited to be here to talk to you about spatial data  exploration in Jupyter notebooks and the path that led me to Jupyter notebooks.  I started working with Python and GIS over 10 years ago when Python support was just  being introduced to QGIS and IG GIS.  And for a long time, particularly QGIS and POST GIS have been my go-to tools for anything  data analysis.  I just went to my blog and dug out some POST from way back 2011 because it basically describes  how I've been working for many, many years.  I did data analysis in QGIS or in POST GIS and afterwards I could, or in between, I could  visualize those results in QGIS, have a look at each step, do the results look okay, can  I continue to the next one and so on, always verifying or testing the hypothesis of the  analysis along the way.  And this was a really important component and still is in all my workflows.  So I even went so far and developed a couple of plugins which you might be aware of for  visualizing particularly movement data in an animated way, so that was the time manager  plugin which has now been superseded by some core functionality in QGIS.  So basically it played into the same need of visually exploring data.  Of course, besides desktop GIS and spatial databases, I've always been aware of other  Python libraries which could be used to write standalone data analysis.  So of course I knew about Shapely, I knew about Fiona, but I rarely used them.  And the reason for that was that they felt kind of redundant for my use.  A lot of functionality that they provided was already included in the POST GIS analysis  functions or in the QGIS API.  So there was no, often no need to add it and also it was quite tricky to install sometimes  it, but the main issue is for me it lacked the visual component that is needed to do  exploratory data analysis.  So I would always have to go to some different tools afterwards to look at the results that  were produced and this would just break my analysis workflow.  This situation has completely changed now, particularly with the introduction of Geopandas  and Jupyter notebooks.  There is now a way to work directly in Python, but also keep this visual component.  However, the default mudplot lib visualizations that are created with Geopandas are still  rather limited.  So you can look at them at their website, on the Geopandas website and you will see they  are quite straightforward, you just call plot and it gives you a map, but it's static.  And by default, there is also no background map to check the geographic context of your  data.  There are some examples in the gallery which show how to add background maps, but they  depend on another library contextily and it's also just a static background map without  any possibility of interaction.  So that was the status quo I was at roughly two years ago and I started looking into,  okay, what are the options for doing it interactively?  How can I actually zoom into the map to see a bit more detail, for example?  And the first library that I looked at was Folium.  I'm sure you've heard of it.  Folium is based on leaflet and it basically allows you to create interactive maps and  put vector data on top.  But I don't want to talk about Folium because that's not where I ended up with.  There I really found my new home and what really works well for me is Holovis and that's  what I want to demo to you today.  And Holovis is not just one library, it's a whole family of libraries and particularly  noteworthy of interest to us today are the GFUs library and the DataShader library.  So they are directly in the name.  GFUs gives you the capabilities to map spatial data and it's very handy and it has a lot of  capabilities like built in background maps and interactive maps that we are going to  explore now in this presentation.  The first example that I want to show you is for all of you who maybe haven't worked  with GeoPandas yet, just a quick look into how you can interact with spatial data and  how simple it is to use.  I'm currently working on a small lab setup which uses open government data which is provided  by the city of Vienna.  So you can have a live demo.  If you click on the link here in binder, it will actually launch this hosted JupyterLab  instance and I will show you exactly what this can do.  I'm just importing a couple of basic libraries here.  You can already see the JupyterLab contextual help updating depending on which module I  select here.  It's super useful help.  But what I really wanted to get into is of course the spatial data analysis component.  So there is a WFS service hosted by the city of Vienna which I'm just going to query.  I'm going to download a GeoChasen file here, particularly of electric vehicle charging  stations.  So basically downloading it here, you can see the Chasen file was downloaded.  And with GeoPandas read file, I can read this GeoChasen file and it gives me a GeoData  frame that I can directly plot here in the notebook.  And you see, I get an interactive map with popups which show me the location and the  designation.  So some name for this station and I can add whichever columns I like to be displayed here  in this one line of code.  So this is what I mean if I say it's really simple to make interactive maps in the notebook  with this library.  Of course GeoPandas also has all kinds of analysis tools or spatial analysis tools.  Here I have just a quick example with buffers and I'm also taking care of coordinate transformations  here.  So here you can see all the stations buffered with a 100 meter buffer.  And if you zoom in, you see the station location itself plus the buffer drawn around it.  And of course I can do that with multiple different data sets.  And the really cool thing is you can plot these data sets then side by side just with  the plus operator here.  And now side by side it's synced.  I can zoom in.  I can compare different neighborhoods and how they are served for example by electric  vehicle charging stations or by city bike stations.  And you can play this as much as you want.  So I've imported another library for Geocoding with Nominatin which is GeoPi and edit the  address of my office and now I can have a look around my office in a buffer of 1 kilometer.  How many charging stations do I have?  Looks like there's two of them.  So this is one basic example which I invite you to visit.  If you just go to OGDAT lab you can launch the binder here.  Obviously this is a very small data set, it comes from this web service directly.  But trust me if I say we are not limited to small data sets.  And the actual larger example that I want to demo you today is about a couple million  data points instead of just a couple dozen.  So let me import the setup for this demo.  Again this is mostly based on GeoPandas and also a lot on HoloViews and DataShader particularly  because that's what we will need to plot the large data sets.  As you can see here based on the code this is an example with AIS data which is Ship  Movement data and I'm importing two days of this data.  Eventually there's a lot of columns so there's also a lot of values which we want to get  rid of because the speed is zero and that's rather boring for most analysis.  So let's read in those two files.  This is going to take a while but I wanted to show it to you how long it takes so that  you can appreciate it later on when we work with the data how quickly it goes.  You can also see here a couple of nice features of Pandas reading from text files.  You can already specify which columns you want to keep, which you want to throw away.  You can rename them so that they have a bit nicer column names to work with later on.  I'm also specifying a temporal index right here.  So I'm turning this dates, time, format strings into temporal objects and assigning them as  the index.  And even though it only shows the date here it's actually a date time so the time information  is not getting lost here.  All in all we have now loaded 12 million records and it's special temporal.  There's an X and a Y column here.  To be worth mentioning I have used a utility function of data shader which converts latitude  and longitude to meters in Psojdomerkarta which then makes it easier to plot them on  a base map.  And that's exactly what we are going to do.  We will want to plot them.  First we can have a look at the basic extent.  Obviously this is rather fast.  It's just pandas accessing the column min and max values.  But now let's plot this.  Let's see how it actually looks like on a map.  And we've now plotted 12 million points on a map and it's interactive.  So we can zoom in.  We can see all the points near Null Island.  Probably some outliers we want to get rid of.  But we can also see lots of points in other areas, our main areas of interest.  We can crop this now.  We can analyze it in all kinds of different ways.  For example we might want to focus only on this small area, this bounding box.  Again pretty fast and interactive.  The default plots of this data shade are actually density maps.  So what's usually called heat maps.  If you compare this example I basically told him to not use a heat map because I used a  color map from blue to blue.  And I did an overwrite of the heat map to just get every individual point to display  here.  But if you don't provide a heat map it will give you this kind of visualization where  you can actually see the density of the points and there you can very easily spot that it's  much denser in this area of northern Europe than in all the other areas where there's  also some data.  Of course since we are in the Python world there's also really good functionality to  do temporal analysis so we can have a look at how many data sets for example we have  per time interval of 15 minutes or whatever you want to specify here.  So in this movement data set you can see that on the 1st of January it's 1st of July it's  super busy and on the 2nd there's much less traffic according to this data set.  And having all these kinds of different analysis in the same notebook obviously has huge advantages  for making it reproducible for documenting the whole data exploration process.  Again you can have multiple plots side by side, you can have multiple maps side by side  like I'm doing here where you can compare two different days.  That'd be very interesting because it looks like the spatial extent of the data set has  changed and on the 2nd we have this whole area here in the west which is basically empty  for some reason.  So it's really cool, it's fast, there is however a couple of downsides so there is no convenient  way to visualize different categories so I can't say ships of ship type A should be  red and ships of ship type B should be blue for example.  Instead the only way that I found so far is to plot multiple scatter plots on top of each  other or next to each other so I can of course filter the whole data frame by different categories  and I can plot them side by side or like shown on the right or on top of each other as shown  on the left but I have to plot each of those filtered data frames individually and I have  to give it a color.  And I cannot do just a one liner which says okay take this data frame and based on this  column apply some chosen colors to each category.  Another feature of data shader that I want to share with you is how you can also work  with line features so until so far we have only plotted point features but obviously  it also can plot line features and it has the same fast rendering which is achieved  by the underlying implementation.  And the example that I want to show you here is how to find segments that have gaps that  represent gaps because there is a huge time difference between consecutive positions.  And what you can see here in the next map once it's drawn is all those segments where  there is a long jump between one location and the follow up location.  And data shader obviously doesn't just work on maps it also works on all kinds of other  plots so if you have a normal scatter plot for example the coordinate changes from one  location to the other you can also plot those and irrespective of the coordinate system  you now have another data shaded plot to explore.  Something also related to this issue with not being able to plot categorized values  is that you cannot visualize the attributes of line strings along the you cannot visualize  the properties of line strings also categorized or graduated.  And so these plots that I have included in the end of this notebook which are showing  you the speed along a line string or along a trajectory those wouldn't be possible as  far as I know with data shader right now so this uses the basic geo-fuse plotting instead  and you can of course also zoom in here and this is vector data that is rendered here as  opposed to the data shaded data which visualizations which are always rasterized in the background  to allow for this quick rasterization.  So sometimes I still have to go back and have to use the basic geo-fuse it's way more convenient  if you want to have access to the attributes and visualize those as well.  So if you're interested in more details on how to analyze movement data I invite you  to visit MovingPanda's on GitHub and go to the example section and one of the examples  that you can find here is a data set from a horse GPS caller that Lena Fischer from  the University of Copenhagen took provided to me and if you go through the examples  you will see a lot of these maps side by side with different backgrounds for example temporal  analysis but what I really wanted to point out is also the capabilities of splitting  the observations into different time frames and then all doing all these evaluations for  example daily trajectory length or the daily covered area and you can see the existing  or non-existing trends of these properties of movement for example and you can also have  a look at to where the moving objects for example spent a certain amount of time which  is called stop detection analysis and always the really neat part and what I want to emphasize  is you just have a couple of lines of code and then there's the visualization of the  intermediate results which should provide you really fast feedback and the possibility  to dive deeper into data than you usually would if this whole workflow was more complicated  to do so definitely ease of use I think makes for better data exploration because you just  get a feeling that you can do more things and on this last note you getting the feeling  of being able to do more things there's one feature that I want to highlight in the end  and that's not just having interactive individual plots that you can zoom in that you can hover  and see feature properties now you can also do linked plots and you can do filtering between  different plots and I want to show this as an example between map plot so again one of  the those scatter plots as we had before and histogram here is an example of a histogram  of speed values so we want to link them together so that we can see for example which speeds  are located where and this is really simple there is just one function that you have to  use which is called link selection and it takes as input the two plots that we just  provided created previously and what you can now do is you can filter here either on the  spatial side and it will update the plot linked on the right side or you can filter here on  the attribute side and it will filter also the geographic side and I think with this  and some of the other tools like sliders and buttons that you can add the possibilities  are basically limitless and on that note I want to encourage you to if you haven't yet  I definitely give it a try I think it's a great addition to your toolbox of if you come  from the more desktop GIS side of things and it certainly is also a great addition to your  toolbox if you come from the classical data science and probably haven't done much yet  with geographic data then I think this is the these are the libraries to get started  with and if you are indeed working with movement data then feel free to reach out and share  your experiences with moving pandas and with any of the notebooks I've shared with you  today and I would be happy to exchange ideas and plans for future directions on these as  well so with that back to you and thank you.
geopandasholoviews_0_2351.png geopandasholoviews_2352_5447.png geopandasholoviews_5448_5879.png geopandasholoviews_5880_6107.png geopandasholoviews_6108_8963.png geopandasholoviews_8964_9095.png geopandasholoviews_9096_14171.png geopandasholoviews_14172_15527.png geopandasholoviews_15528_18539.png geopandasholoviews_18540_18647.png geopandasholoviews_18648_18743.png geopandasholoviews_18744_19343.png geopandasholoviews_19344_20243.png geopandasholoviews_20244_20795.png geopandasholoviews_20796_21791.png geopandasholoviews_21792_21851.png geopandasholoviews_21852_24875.png geopandasholoviews_24876_28235.png geopandasholoviews_28236_29267.png geopandasholoviews_29268_29315.png geopandasholoviews_29316_29543.png geopandasholoviews_29544_29639.png geopandasholoviews_29640_29891.png geopandasholoviews_29892_31151.png geopandasholoviews_31152_35613.png
Welcome, my name is Anita Grazer. This is my first FOSDEM, so I'm really excited to be here to talk to you about spatial data exploration in Jupyter notebooks and the path that led me to Jupyter notebooks. I started working with Python and GIS over 10 years ago when Python support was just being introduced to QGIS and IG GIS. And for a long time, particularly QGIS and POST GIS have been my go-to tools for anything data analysis. I just went to my blog and dug out some POST from way back 2011 because it basically describes how I've been working for many, many years. I did data analysis in QGIS or in POST GIS and afterwards I could, or in between, I could visualize those results in QGIS, have a look at each step, do the results look okay, can I continue to the next one and so on, always verifying or testing the hypothesis of the analysis along the way. And this was a really important component and still is in all my workflows. So I even went so far and developed a couple of plugins which you might be aware of for visualizing particularly movement data in an animated way, so that was the time manager plugin which has now been superseded by some core functionality in QGIS. So basically it played into the same need of visually exploring data. Of course, besides desktop GIS and spatial databases, I've always been aware of other Python libraries which could be used to write standalone data analysis. So of course I knew about Shapely, I knew about Fiona, but I rarely used them. And the reason for that was that they felt kind of redundant for my use. A lot of functionality that they provided was already included in the POST GIS analysis functions or in the QGIS API. So there was no, often no need to add it and also it was quite tricky to install sometimes it, but the main issue is for me it lacked the visual component that is needed to do exploratory data analysis. So I would always have to go to some different tools afterwards to look at the results that were produced and this would just break my analysis workflow. This situation has completely changed now, particularly with the introduction of Geopandas and Jupyter notebooks. There is now a way to work directly in Python, but also keep this visual component. However, the default mudplot lib visualizations that are created with Geopandas are still rather limited. So you can look at them at their website, on the Geopandas website and you will see they are quite straightforward, you just call plot and it gives you a map, but it's static. And by default, there is also no background map to check the geographic context of your data. There are some examples in the gallery which show how to add background maps, but they depend on another library contextily and it's also just a static background map without any possibility of interaction. So that was the status quo I was at roughly two years ago and I started looking into, okay, what are the options for doing it interactively? How can I actually zoom into the map to see a bit more detail, for example? And the first library that I looked at was Folium. I'm sure you've heard of it. Folium is based on leaflet and it basically allows you to create interactive maps and put vector data on top. But I don't want to talk about Folium because that's not where I ended up with. There I really found my new home and what really works well for me is Holovis and that's what I want to demo to you today. And Holovis is not just one library, it's a whole family of libraries and particularly noteworthy of interest to us today are the GFUs library and the DataShader library. So they are directly in the name. GFUs gives you the capabilities to map spatial data and it's very handy and it has a lot of capabilities like built in background maps and interactive maps that we are going to explore now in this presentation. The first example that I want to show you is for all of you who maybe haven't worked with GeoPandas yet, just a quick look into how you can interact with spatial data and how simple it is to use. I'm currently working on a small lab setup which uses open government data which is provided by the city of Vienna. So you can have a live demo. If you click on the link here in binder, it will actually launch this hosted JupyterLab instance and I will show you exactly what this can do. I'm just importing a couple of basic libraries here. You can already see the JupyterLab contextual help updating depending on which module I select here. It's super useful help. But what I really wanted to get into is of course the spatial data analysis component. So there is a WFS service hosted by the city of Vienna which I'm just going to query. I'm going to download a GeoChasen file here, particularly of electric vehicle charging stations. So basically downloading it here, you can see the Chasen file was downloaded. And with GeoPandas read file, I can read this GeoChasen file and it gives me a GeoData frame that I can directly plot here in the notebook. And you see, I get an interactive map with popups which show me the location and the designation. So some name for this station and I can add whichever columns I like to be displayed here in this one line of code. So this is what I mean if I say it's really simple to make interactive maps in the notebook with this library. Of course GeoPandas also has all kinds of analysis tools or spatial analysis tools. Here I have just a quick example with buffers and I'm also taking care of coordinate transformations here. So here you can see all the stations buffered with a 100 meter buffer. And if you zoom in, you see the station location itself plus the buffer drawn around it. And of course I can do that with multiple different data sets. And the really cool thing is you can plot these data sets then side by side just with the plus operator here. And now side by side it's synced. I can zoom in. I can compare different neighborhoods and how they are served for example by electric vehicle charging stations or by city bike stations. And you can play this as much as you want. So I've imported another library for Geocoding with Nominatin which is GeoPi and edit the address of my office and now I can have a look around my office in a buffer of 1 kilometer. How many charging stations do I have? Looks like there's two of them. So this is one basic example which I invite you to visit. If you just go to OGDAT lab you can launch the binder here. Obviously this is a very small data set, it comes from this web service directly. But trust me if I say we are not limited to small data sets. And the actual larger example that I want to demo you today is about a couple million data points instead of just a couple dozen. So let me import the setup for this demo. Again this is mostly based on GeoPandas and also a lot on HoloViews and DataShader particularly because that's what we will need to plot the large data sets. As you can see here based on the code this is an example with AIS data which is Ship Movement data and I'm importing two days of this data. Eventually there's a lot of columns so there's also a lot of values which we want to get rid of because the speed is zero and that's rather boring for most analysis. So let's read in those two files. This is going to take a while but I wanted to show it to you how long it takes so that you can appreciate it later on when we work with the data how quickly it goes. You can also see here a couple of nice features of Pandas reading from text files. You can already specify which columns you want to keep, which you want to throw away. You can rename them so that they have a bit nicer column names to work with later on. I'm also specifying a temporal index right here. So I'm turning this dates, time, format strings into temporal objects and assigning them as the index. And even though it only shows the date here it's actually a date time so the time information is not getting lost here. All in all we have now loaded 12 million records and it's special temporal. There's an X and a Y column here. To be worth mentioning I have used a utility function of data shader which converts latitude and longitude to meters in Psojdomerkarta which then makes it easier to plot them on a base map. And that's exactly what we are going to do. We will want to plot them. First we can have a look at the basic extent. Obviously this is rather fast. It's just pandas accessing the column min and max values. But now let's plot this. Let's see how it actually looks like on a map. And we've now plotted 12 million points on a map and it's interactive. So we can zoom in. We can see all the points near Null Island. Probably some outliers we want to get rid of. But we can also see lots of points in other areas, our main areas of interest. We can crop this now. We can analyze it in all kinds of different ways. For example we might want to focus only on this small area, this bounding box. Again pretty fast and interactive. The default plots of this data shade are actually density maps. So what's usually called heat maps. If you compare this example I basically told him to not use a heat map because I used a color map from blue to blue. And I did an overwrite of the heat map to just get every individual point to display here. But if you don't provide a heat map it will give you this kind of visualization where you can actually see the density of the points and there you can very easily spot that it's much denser in this area of northern Europe than in all the other areas where there's also some data. Of course since we are in the Python world there's also really good functionality to do temporal analysis so we can have a look at how many data sets for example we have per time interval of 15 minutes or whatever you want to specify here. So in this movement data set you can see that on the 1st of January it's 1st of July it's super busy and on the 2nd there's much less traffic according to this data set. And having all these kinds of different analysis in the same notebook obviously has huge advantages for making it reproducible for documenting the whole data exploration process. Again you can have multiple plots side by side, you can have multiple maps side by side like I'm doing here where you can compare two different days. That'd be very interesting because it looks like the spatial extent of the data set has changed and on the 2nd we have this whole area here in the west which is basically empty for some reason. So it's really cool, it's fast, there is however a couple of downsides so there is no convenient way to visualize different categories so I can't say ships of ship type A should be red and ships of ship type B should be blue for example. Instead the only way that I found so far is to plot multiple scatter plots on top of each other or next to each other so I can of course filter the whole data frame by different categories and I can plot them side by side or like shown on the right or on top of each other as shown on the left but I have to plot each of those filtered data frames individually and I have to give it a color. And I cannot do just a one liner which says okay take this data frame and based on this column apply some chosen colors to each category. Another feature of data shader that I want to share with you is how you can also work with line features so until so far we have only plotted point features but obviously it also can plot line features and it has the same fast rendering which is achieved by the underlying implementation. And the example that I want to show you here is how to find segments that have gaps that represent gaps because there is a huge time difference between consecutive positions. And what you can see here in the next map once it's drawn is all those segments where there is a long jump between one location and the follow up location. And data shader obviously doesn't just work on maps it also works on all kinds of other plots so if you have a normal scatter plot for example the coordinate changes from one location to the other you can also plot those and irrespective of the coordinate system you now have another data shaded plot to explore. Something also related to this issue with not being able to plot categorized values is that you cannot visualize the attributes of line strings along the you cannot visualize the properties of line strings also categorized or graduated. And so these plots that I have included in the end of this notebook which are showing you the speed along a line string or along a trajectory those wouldn't be possible as far as I know with data shader right now so this uses the basic geo-fuse plotting instead and you can of course also zoom in here and this is vector data that is rendered here as opposed to the data shaded data which visualizations which are always rasterized in the background to allow for this quick rasterization. So sometimes I still have to go back and have to use the basic geo-fuse it's way more convenient if you want to have access to the attributes and visualize those as well. So if you're interested in more details on how to analyze movement data I invite you to visit MovingPanda's on GitHub and go to the example section and one of the examples that you can find here is a data set from a horse GPS caller that Lena Fischer from the University of Copenhagen took provided to me and if you go through the examples you will see a lot of these maps side by side with different backgrounds for example temporal analysis but what I really wanted to point out is also the capabilities of splitting the observations into different time frames and then all doing all these evaluations for example daily trajectory length or the daily covered area and you can see the existing or non-existing trends of these properties of movement for example and you can also have a look at to where the moving objects for example spent a certain amount of time which is called stop detection analysis and always the really neat part and what I want to emphasize is you just have a couple of lines of code and then there's the visualization of the intermediate results which should provide you really fast feedback and the possibility to dive deeper into data than you usually would if this whole workflow was more complicated to do so definitely ease of use I think makes for better data exploration because you just get a feeling that you can do more things and on this last note you getting the feeling of being able to do more things there's one feature that I want to highlight in the end and that's not just having interactive individual plots that you can zoom in that you can hover and see feature properties now you can also do linked plots and you can do filtering between different plots and I want to show this as an example between map plot so again one of the those scatter plots as we had before and histogram here is an example of a histogram of speed values so we want to link them together so that we can see for example which speeds are located where and this is really simple there is just one function that you have to use which is called link selection and it takes as input the two plots that we just provided created previously and what you can now do is you can filter here either on the spatial side and it will update the plot linked on the right side or you can filter here on the attribute side and it will filter also the geographic side and I think with this and some of the other tools like sliders and buttons that you can add the possibilities are basically limitless and on that note I want to encourage you to if you haven't yet I definitely give it a try I think it's a great addition to your toolbox of if you come from the more desktop GIS side of things and it certainly is also a great addition to your toolbox if you come from the classical data science and probably haven't done much yet with geographic data then I think this is the these are the libraries to get started with and if you are indeed working with movement data then feel free to reach out and share your experiences with moving pandas and with any of the notebooks I've shared with you today and I would be happy to exchange ideas and plans for future directions on these as well so with that back to you and thank you.
visualizing particularly movement data in an basically allows you to create interactive maps family of libraries and particularly noteworthy government data which is provided by the city of from this web service directly. But trust me if I Of course since we are in the Python world visualize those as well. So if you're interested the possibility to dive deeper into data than you