Push it through the wire! Push it more, if it's wireless!
Cepicky, Jachym
Today's web browsers, their rendering engines and JavaScript interpreters are able to display relatively big amounts of vector data. Moving from DOM rendering (as it was implemented with help of SVG in for examples OpenLayers 2) to Canvas (and further to WebGL - as we are now having in OpenLayers 3 or Leaflet) enables us to display thousands of complex vector features, with complicated on-client vector data styling. With this possibility, we are facing now new types problems: how to send such amount of data through limited internet connection? If we have closer look at the problem, we can see clearly, that old database paradigm has raised one more time: we can not have all three attributes of data in one pot, but only 2 of them: speed of the delivered data or amount of delivered data or their topicality. If we take this limits into account and decide to deal with big amounts of data in fast way, topicality must be sacrificed. In the talk, we will demonstrate some possible solutions for this problem, using tiled vectors, generalization, aggregation of vector data. Also advantages, disadvantages of various new and popular vector formats, such as GeoJSON, TopoJSON or MapBox will be discussed. Geometric data do not have be rendered all the time in all scales and over whole area of interest, but only necessary portion of them. If displayed in smaller scales, aggregation and generalisation can take place on the server side. That implies, that using vector caching mechanism could be considered as well. But if we need direct interaction of the server input with cached vector data, mechanism for this must be defined as well. Also attribute data have to be transfered separately, if all the optimisation was put in the vector geometries. Also possible steps between cached data and real-time data will be discussed.
 Better. Okay. Hi. My name is Jachim Cepitski. I welcome to my talk about how to get data through wireless or even wireless. I have been involved longer than 10 years with development of open source software for geospatial, especially. I started on the desktop side using grad or developing for and with grads GIS. Then I slowly moved to server side and last years, like a couple of years I make my living, I do make my living through development of JavaScript based web, GIS is web applications, whatever. What is this? I have an error here. Okay. You probably know that the development of web based applications has completely changed during last couple of years. We are no longer writing JavaScript code and sending it somehow directly to the client side, like we did in the early days. This is nice. Okay. When I set up the presentation, I check the first slides. They seem to be working. And now, as you may see, there is like nothing. I'm sorry. No, I didn't copy it to the hard drive. I see. Thank you. Okay. Thank you for the link. Nice. Okay. Sorry. Another start. I will have to move faster. Okay. Next. Yeah. Another try. Yeah. Cool. Works. Yeah. So we are no longer sending JavaScript files to the browser. Instead, we are compiling the code again. The code is running faster. We transfer not so many data, not so much the code, the program itself isn't that big. You can compress, for example, code of open layers 3 right now from more than a megabyte than several kilobytes for concrete example. The other side, this is what we are facing right now, or at least a team I am working with. The compilation of the whole application takes like several minutes, and if you are deploying it, and if you won't have continuous integration, it takes really a couple of minutes, so to say, or maybe an hour or two a day to get code compiled. Yep. But nowadays, JavaScript renders and interpreters, they do not have a problem with rendering. Really, really big amounts of data which we could see in the presentations during the whole FOS4G, not only FOS4G. And raster images with them is pretty simply, they are coming with compressed form using JPEGs or PNG files. And for rendering a raster image, there is no rasterization needed anymore, actually, because at the end, there finally is somewhere at the end, you have to transfer the data from the vector to raster, of course, in order to be able to display them. The amount of the information in the raster file can be significantly lower if you compare it to vector file. We can say that the amount of information in a raster file is, of course, limited to its resolution and number of colors you are able to somehow to transfer to the user, so the user can really still interpret it. But the amount of information in a vector file is given by, so to say, number of vertexes, you need for description of a feature, like for example, adoc. And, of course, the number of attributes you are able to put into your vector data. From certain point of, from certain level of detail, the size of a vector data might be bigger, actually, than the size of raster data, though. Why would you need the scope of this presentation is how to transfer vector data to the client? And why would you need this anyway, right? Anyhow. You want the user to enable it, interact with the data. There are dynamic maps. You want to somehow distribute the dynamic aspect of the data. Of course, you want to enable the user to edit the data and more other reasons. But the main reason why I think it's the most important one is that vector data are simply cool. And it gives you all the power of having the data on client side. You can do whatever you need to do with them. But we are facing here the common database parallel. You cannot have all three attributes of the data at once. Even if you keep trying, if you keep pushing, whatever you try to do, you always have to struggle. Either you will have big data or you will deliver the data in a fast way, or the data will be up to date. You can always just do two of those attributes, but never all three of them, whatever you try to do. Those would be the suggested steps when you deal with large amount of data. Of course, of vector data. Various strategies, so to say. It starts from the beginning, from the very simplest one, and then it goes through the most complicated one. The first step is compress the communication between the server and the client. Then the other, choose a proper file format, actually, for the data transfer itself. Use cache, obviously. Use lossy compression for vector data as well. You can do this with rosters, why not for vectors? Transfer only the data, the user really, really needs to see, or the application really, really needs to have. To behave in some way that the user has a feeling of having everything at hand. The first step, compress server-client communication. It might be obvious, right? But you would be probably surprised how often the system administrators forget the simplest thing. Switch on, for example, mod deflate on Apache server. If you take an 80 megabytes big geogson file, it contains 50,000 polygons based on OpenStreetPan data, you can really make it 10 times smaller. Just like that, just with single configuration option. So don't forget this. This is probably the first start. If something is slow, you have a feeling that there is something slow. This would be probably the first check I would do. The other check would be, do I use the proper file format? Everybody is talking about geogson, right, nowadays. But surprisingly, to me, it was a really big surprise. Geogson isn't the most compressed format for vector data. If you compare it to GML, surprisingly, it's twice that big. Again, I'm speaking about 50,000 of polygons based on OpenStreetPan data, randomly picked from the dataset. I don't know actually why this is possible. I have rechecked several times. I have a feeling that the GML file doesn't contain attributes where there is no value, while geogson had really, really all the attributes, even though they were empty, I think so. I just used standard OGR library for data transformation. But on the other hand, of course, file size isn't everything. You can imagine parsing 40 megabytes GML in the browser, takes a certain amount of energy and processor time compared to geogson as well. What I want to say, just be aware of that. Use cache, another strategy. The cache doesn't necessarily have to be, you don't have to set up the whole tile cache stack. Might be enough if you turn on, for example, net framework cache, if you are using net framework for your coding or whatever. And you simply cache just a little amount of your data on the application on the server side, so that the application doesn't have to really touch the database. If nothing has changed, there is no reason for touching the database. And if you want to deliver the data in a fast way, and for some reason, might be data size, might be badly optimization, or badly indexed files in your post-GIS database, whatever. If you turn on cache on your application server, it makes it work really, really much faster than if you go to the data directly. And again, it doesn't cost you basically anything, it's just a few lines of code. And you can have a fast result, actually, very easily. Okay, another strategy. If you are displaying compressed JPEG files, for example, and for users, and everybody knows that it looks nearly as the reality, but users don't complain, it's good enough for them. So why couldn't we do the same with vectors? I was already mentioned here, as was for G as well. Who needs actually 1000 of vertexes per line for, of course, certain zoom level? When, for example, 3 vertexes or 4 would be enough for the data line, or polygon, or whatever representation. Backdoor mapping libraries, the modern ones, like OpenLayers, or OpenLayers 3, of course, or Leaflet, they do actually have a pre-processing of the data, which are at hand, before they render it in, for example, canvas render. So they are really then the most expensive part, the rendering itself. It's using, it's working with really a compressed number of vertexes, if you are rendering lines or polygons. But, of course, that doesn't solve the problem. It's not that you need the data to be transferred to the client. This is everything happening just on the client side, based on the whole dataset. But I would advise you to have a look at the application over there, where you can really play set number of vertexes. You need to perline and see what's happening there. And this is what's happening on the client side, somehow, in the background. One of the formats which actually implements the compression is TopoJSON format. It originates on JoeJSON, and again, my concrete example, if I take 80 megabytes JoeJSON polygons, I convert it to TopoJSON. It has only 3 megabytes, and if I zip it using Modiflate, then it has only 50 kilobytes. It seemed to be magic to me. I would like somebody in the audience to reproduce such experiments, because it seems impossible, but if you render the dataset, it really looks like it's supposed to be. And, yeah, with just a little bit of data processing, you can get pretty good results. TopoJSON was introduced by Bostock. Everybody knows TopoJSON. I think it doesn't make sense to talk much further about it, for one reason that I don't really understand what's behind it, what's behind the magic, except for, yeah, it uses relative coordinates, delta values from some starting point, so to say. And there is an internal index of arcs, so-called and so on. There is a link to documentation on GitHub where you can get into the detail. Sorry, you wanted to make a picture? Okay, the presentation will be online later. Okay, and as I said, it uses less compression, since the delta values are integer values, basically. So you really don't, you lose a certain level of detail, but for most of the applications, it's far enough. But there's one issue I was facing with TopoJSON. There is no support for so-called big integer numbers, which again shouldn't be a problem for 99% of your use cases. But my use case is I want to display a catastrophic parcel data in the web browser, and the RD is big integer then. Of course, there's a workaround, you know, you can create your own IDs and use string, for example, for this one, but yeah, it isn't that nice. And one of the next steps would be you can, you don't have to transfer it, or you don't want to transfer all the data, which are at hand, in one step from the server to the client. So till now we were speaking about data shrinking, but now another strategy, so that the user has a feeling that the application is somehow living, that the data are there, even though they might be coming in several steps. You don't have to transfer everything in one step. We are used to tiled restores, and everybody is speaking about tiled vectors. There were several talks already about tiled vectors. This concrete example is based on openlers, and it uses actually JSON tiles. Okay, another five minutes. It's relatively complicated to get them rendered compared to restores, of course, but nowadays libraries, they support it. It's basically no problem, so what you can see here in the presentation, oh, I can. Great. Is this border between those tiles, but this was a bucket basically in openers and it's gone, so there's no problem with that. Then you need to set up the whole infrastructure that makes things a little bit complicated. One of the possible solutions would be tiles-tiles-project and Python-based project, which generates you, which is able not only to generate raster tiles, but also geo-Json tiles and even topo-Json tiles, which is very nice. I was facing some difficulties using different projection than spherical mercators and Google Mercator or Web Mercator projection. But that was the only geo-ahead with that, otherwise it works nicely. Another standard is mapbox-tiles. Again, we were addressing this during the whole FOS4G. Mapbox has developed and opened a standard for basically binary vector encoding. The first thing is that the coordinates of the data do not use the geographical coordinates anymore, it uses relative coordinates of the image or of the whole map. So rendering calculations, there's no calculations, it renders pretty fast. And of course it's supported in leaflet. In openers, it's going to be supported soon. Soon means that I already tested it on my computer, it's in a branch and it will be there soon. And for generation, you don't have to use mapbox infrastructure, you can use geo-server or again in a very near future map server. I saw it. There might be another strategy you might take. If you have a look at this polygon, apparently it is split in this case and to four different vector features. And it is really like it is in your map application, you will have at least four different features with the same ID, for example. Then if you provide some filtering, some selections, of course the application has to be adjusted to that fact. Another option would be that you download just the vector data, which for example, centroids fit into particle tile. Then it could look like this. You see there are no clear boundaries because I'm downloading whole vector features, which centroid fit into the virtual. So to say with two running box of the tile. Of course, if you have some long power source, long polygon data, for example, or long point data on the edge of the tile, then the vector feature won't be loaded. It depends. But for most of these cases, this seems to be working pretty well. And then you have only one feature, one times as one object in your application. Yes, and result of this effort around it. And user will see just what he or she is expecting when you are dealing with raster tiles. That means there will be somehow loading tiles. The map will slowly appear and the user will have a feeling that something is happening, which is always good. Conclusion. The first question would be, do I need to use all this compression file format, tiling, stack, when I want to display single GPS log file? Answer is obvious. Those approaches are just for situation when the application seems to be slow for some reason. If you are trying to load some bigger amount of data and you have to define the bottlenecks and, yeah, those would be the strategies you can take if you want to reduce the time, the download time, or the rendering time, whatever. Caching and tiling of vector data gives you more speed, but of course you always can have only two attributes out of three. And with caching, you lose the timeliness of the data, because there will always be some cache. For some kind of application, this is perfectly okay. Everybody can live with Google data even though everybody knows they are pre-cached. And you never get the same result asking from different side of the world, asking Google. Or, yeah, you never get the same result on two parts of the world at the same time, because the databases isn't synchronized and so on. They are not up to date. But it's fine. Good enough. Thank you for your attention. Any questions? Yeah, as I said, we have like five minutes, two minutes for questions. So one or two. Okay. Yeah, please. Okay, you need the mic. The mic is running. So you did try to plug OpenLayers to Mapserver? Yes, I did. Yes, it works. We did it just yesterday together with Thomas Monfort and Andrea Sihokover, one from the Mapserver side, the other one from the OpenLayers side, with myData. MyData means really reliving catastrophe data, partial data, and in PostGIS database. I wanted to show it, but I wasn't able to plug my computer into this VGA output. So I can show it to you later, separate and private. Okay, thank you. Then I would like to ask the next speaker.
 Better.  Okay.  Hi.  My name is Jachim Cepitski.  I welcome to my talk about how to get data through wireless or even wireless.  I have been involved longer than 10 years with development of open source software for geospatial,  especially.  I started on the desktop side using grad or developing for and with grads GIS.  Then I slowly moved to server side and last years, like a couple of years I make my living,  I do make my living through development of JavaScript based web, GIS is web applications, whatever.  What is this?  I have an error here.  Okay. You probably know that the development of web based applications has completely changed  during last couple of years.  We are no longer writing JavaScript code and sending it somehow directly to the client side,  like we did in the early days.  This is nice.  Okay. When I set up the presentation, I check the first slides.  They seem to be working.  And now, as you may see, there is like nothing.  I'm sorry.  No, I didn't copy it to the hard drive.  I see. Thank you.  Okay. Thank you for the link.  Nice.  Okay. Sorry. Another start. I will have to move faster.  Okay.  Next.  Yeah. Another try.  Yeah. Cool. Works.  Yeah. So we are no longer sending JavaScript files to the browser.  Instead, we are compiling the code again.  The code is running faster.  We transfer not so many data, not so much the code, the program itself isn't that big.  You can compress, for example, code of open layers 3 right now from more than a megabyte  than several kilobytes for concrete example.  The other side, this is what we are facing right now, or at least a team I am working with.  The compilation of the whole application takes like several minutes, and if you are deploying it,  and if you won't have continuous integration, it takes really a couple of minutes, so to say,  or maybe an hour or two a day to get code compiled.  Yep.  But nowadays, JavaScript renders and interpreters,  they do not have a problem with rendering.  Really, really big amounts of data which we could see in the presentations during the whole FOS4G,  not only FOS4G.  And raster images with them is pretty simply, they are coming with compressed form using JPEGs or PNG files.  And for rendering a raster image, there is no rasterization needed anymore, actually,  because at the end, there finally is somewhere at the end, you have to transfer the data from the vector to raster, of course,  in order to be able to display them.  The amount of the information in the raster file can be significantly lower if you compare it to vector file.  We can say that the amount of information in a raster file is, of course, limited to its resolution  and number of colors you are able to somehow to transfer to the user,  so the user can really still interpret it.  But the amount of information in a vector file is given by, so to say, number of vertexes,  you need for description of a feature, like for example, adoc.  And, of course, the number of attributes you are able to put into your vector data.  From certain point of, from certain level of detail, the size of a vector data might be bigger, actually,  than the size of raster data, though.  Why would you need the scope of this presentation is how to transfer vector data to the client?  And why would you need this anyway, right? Anyhow.  You want the user to enable it, interact with the data.  There are dynamic maps. You want to somehow distribute the dynamic aspect of the data.  Of course, you want to enable the user to edit the data and more other reasons.  But the main reason why I think it's the most important one is that vector data are simply cool.  And it gives you all the power of having the data on client side.  You can do whatever you need to do with them.  But we are facing here the common database parallel.  You cannot have all three attributes of the data at once.  Even if you keep trying, if you keep pushing, whatever you try to do,  you always have to struggle.  Either you will have big data or you will deliver the data in a fast way,  or the data will be up to date.  You can always just do two of those attributes, but never all three of them, whatever you try to do.  Those would be the suggested steps when you deal with large amount of data.  Of course, of vector data.  Various strategies, so to say.  It starts from the beginning, from the very simplest one, and then it goes through the most complicated one.  The first step is compress the communication between the server and the client.  Then the other, choose a proper file format, actually, for the data transfer itself.  Use cache, obviously.  Use lossy compression for vector data as well.  You can do this with rosters, why not for vectors?  Transfer only the data, the user really, really needs to see, or the application really, really needs to have.  To behave in some way that the user has a feeling of having everything at hand.  The first step, compress server-client communication.  It might be obvious, right?  But you would be probably surprised how often the system administrators forget the simplest thing.  Switch on, for example, mod deflate on Apache server.  If you take an 80 megabytes big geogson file, it contains 50,000 polygons based on OpenStreetPan data,  you can really make it 10 times smaller.  Just like that, just with single configuration option.  So don't forget this.  This is probably the first start.  If something is slow, you have a feeling that there is something slow.  This would be probably the first check I would do.  The other check would be, do I use the proper file format?  Everybody is talking about geogson, right, nowadays.  But surprisingly, to me, it was a really big surprise.  Geogson isn't the most compressed format for vector data.  If you compare it to GML, surprisingly, it's twice that big.  Again, I'm speaking about 50,000 of polygons based on OpenStreetPan data,  randomly picked from the dataset.  I don't know actually why this is possible.  I have rechecked several times.  I have a feeling that the GML file doesn't contain attributes  where there is no value, while geogson had really, really all the attributes,  even though they were empty, I think so.  I just used standard OGR library for data transformation.  But on the other hand, of course, file size isn't everything.  You can imagine parsing 40 megabytes GML in the browser,  takes a certain amount of energy and processor time compared to geogson as well.  What I want to say, just be aware of that.  Use cache, another strategy.  The cache doesn't necessarily have to be, you don't have to set up the whole tile cache stack.  Might be enough if you turn on, for example, net framework cache,  if you are using net framework for your coding or whatever.  And you simply cache just a little amount of your data on the application on the server side,  so that the application doesn't have to really touch the database.  If nothing has changed, there is no reason for touching the database.  And if you want to deliver the data in a fast way,  and for some reason, might be data size, might be badly optimization,  or badly indexed files in your post-GIS database, whatever.  If you turn on cache on your application server,  it makes it work really, really much faster than if you go to the data directly.  And again, it doesn't cost you basically anything, it's just a few lines of code.  And you can have a fast result, actually, very easily.  Okay, another strategy.  If you are displaying compressed JPEG files, for example, and for users,  and everybody knows that it looks nearly as the reality, but users don't complain,  it's good enough for them.  So why couldn't we do the same with vectors?  I was already mentioned here, as was for G as well.  Who needs actually 1000 of vertexes per line for, of course, certain zoom level?  When, for example, 3 vertexes or 4 would be enough for the data line,  or polygon, or whatever representation.  Backdoor mapping libraries, the modern ones, like OpenLayers,  or OpenLayers 3, of course, or Leaflet, they do actually have a pre-processing of the data,  which are at hand, before they render it in, for example, canvas render.  So they are really then the most expensive part, the rendering itself.  It's using, it's working with really a compressed number of vertexes,  if you are rendering lines or polygons.  But, of course, that doesn't solve the problem.  It's not that you need the data to be transferred to the client.  This is everything happening just on the client side, based on the whole dataset.  But I would advise you to have a look at the application over there,  where you can really play set number of vertexes.  You need to perline and see what's happening there.  And this is what's happening on the client side, somehow, in the background.  One of the formats which actually implements the compression is TopoJSON format.  It originates on JoeJSON, and again, my concrete example,  if I take 80 megabytes JoeJSON polygons, I convert it to TopoJSON.  It has only 3 megabytes, and if I zip it using Modiflate, then it has only 50 kilobytes.  It seemed to be magic to me.  I would like somebody in the audience to reproduce such experiments,  because it seems impossible, but if you render the dataset, it really looks like it's supposed to be.  And, yeah, with just a little bit of data processing, you can get pretty good results.  TopoJSON was introduced by Bostock. Everybody knows TopoJSON.  I think it doesn't make sense to talk much further about it,  for one reason that I don't really understand what's behind it, what's behind the magic, except for, yeah,  it uses relative coordinates, delta values from some starting point, so to say.  And there is an internal index of arcs, so-called and so on.  There is a link to documentation on GitHub where you can get into the detail.  Sorry, you wanted to make a picture?  Okay, the presentation will be online later.  Okay, and as I said, it uses less compression, since the delta values are integer values, basically.  So you really don't, you lose a certain level of detail, but for most of the applications, it's far enough.  But there's one issue I was facing with TopoJSON.  There is no support for so-called big integer numbers, which again shouldn't be a problem for 99% of your use cases.  But my use case is I want to display a catastrophic parcel data in the web browser, and the RD is big integer then.  Of course, there's a workaround, you know, you can create your own IDs and use string, for example, for this one, but yeah, it isn't that nice.  And one of the next steps would be you can, you don't have to transfer it, or you don't want to transfer all the data, which are at hand,  in one step from the server to the client.  So till now we were speaking about data shrinking, but now another strategy, so that the user has a feeling that the application is somehow living,  that the data are there, even though they might be coming in several steps.  You don't have to transfer everything in one step.  We are used to tiled restores, and everybody is speaking about tiled vectors.  There were several talks already about tiled vectors.  This concrete example is based on openlers, and it uses actually JSON tiles.  Okay, another five minutes.  It's relatively complicated to get them rendered compared to restores, of course, but nowadays libraries, they support it.  It's basically no problem, so what you can see here in the presentation, oh, I can.  Great.  Is this border between those tiles, but this was a bucket basically in openers and it's gone, so there's no problem with that.  Then you need to set up the whole infrastructure that makes things a little bit complicated.  One of the possible solutions would be tiles-tiles-project and Python-based project, which generates you, which is able not only to generate raster tiles,  but also geo-Json tiles and even topo-Json tiles, which is very nice.  I was facing some difficulties using different projection than spherical mercators and Google Mercator or Web Mercator projection.  But that was the only geo-ahead with that, otherwise it works nicely.  Another standard is mapbox-tiles.  Again, we were addressing this during the whole FOS4G.  Mapbox has developed and opened a standard for basically binary vector encoding.  The first thing is that the coordinates of the data do not use the geographical coordinates anymore, it uses relative coordinates of the image or of the whole map.  So rendering calculations, there's no calculations, it renders pretty fast.  And of course it's supported in leaflet.  In openers, it's going to be supported soon.  Soon means that I already tested it on my computer, it's in a branch and it will be there soon.  And for generation, you don't have to use mapbox infrastructure, you can use geo-server or again in a very near future map server.  I saw it.  There might be another strategy you might take.  If you have a look at this polygon, apparently it is split in this case and to four different vector features.  And it is really like it is in your map application, you will have at least four different features with the same ID, for example.  Then if you provide some filtering, some selections, of course the application has to be adjusted to that fact.  Another option would be that you download just the vector data, which for example, centroids fit into particle tile.  Then it could look like this.  You see there are no clear boundaries because I'm downloading whole vector features, which centroid fit into the virtual.  So to say with two running box of the tile.  Of course, if you have some long power source, long polygon data, for example, or long point data on the edge of the tile, then the vector feature won't be loaded.  It depends.  But for most of these cases, this seems to be working pretty well.  And then you have only one feature, one times as one object in your application.  Yes, and result of this effort around it.  And user will see just what he or she is expecting when you are dealing with raster tiles.  That means there will be somehow loading tiles.  The map will slowly appear and the user will have a feeling that something is happening, which is always good.  Conclusion.  The first question would be, do I need to use all this compression file format, tiling, stack, when I want to display single GPS log file?  Answer is obvious.  Those approaches are just for situation when the application seems to be slow for some reason.  If you are trying to load some bigger amount of data and you have to define the bottlenecks and, yeah, those would be the strategies you can take if you want to reduce the time, the download time, or the rendering time, whatever.  Caching and tiling of vector data gives you more speed, but of course you always can have only two attributes out of three.  And with caching, you lose the timeliness of the data, because there will always be some cache.  For some kind of application, this is perfectly okay.  Everybody can live with Google data even though everybody knows they are pre-cached.  And you never get the same result asking from different side of the world, asking Google.  Or, yeah, you never get the same result on two parts of the world at the same time, because the databases isn't synchronized and so on.  They are not up to date.  But it's fine. Good enough.  Thank you for your attention.  Any questions?  Yeah, as I said, we have like five minutes, two minutes for questions.  So one or two.  Okay. Yeah, please.  Okay, you need the mic.  The mic is running.  So you did try to plug OpenLayers to Mapserver?  Yes, I did.  Yes, it works.  We did it just yesterday together with Thomas Monfort and Andrea Sihokover, one from the Mapserver side, the other one from the OpenLayers side, with myData.  MyData means really reliving catastrophe data, partial data, and in PostGIS database.  I wanted to show it, but I wasn't able to plug my computer into this VGA output.  So I can show it to you later, separate and private.  Okay, thank you. Then I would like to ask the next speaker.
PT8-14_0_95.png PT8-14_96_383.png PT8-14_384_1127.png PT8-14_1128_1295.png PT8-14_1296_1487.png PT8-14_1488_1739.png PT8-14_1740_2123.png PT8-14_2124_4067.png PT8-14_4068_4223.png PT8-14_4224_4295.png PT8-14_4296_4391.png PT8-14_4392_4547.png PT8-14_4548_5603.png PT8-14_5604_5675.png PT8-14_5676_7619.png PT8-14_7620_12515.png PT8-14_12516_14651.png PT8-14_14652_16355.png PT8-14_16356_16919.png PT8-14_16920_18947.png PT8-14_18948_24647.png PT8-14_24648_25187.png PT8-14_25188_25271.png PT8-14_25272_25547.png PT8-14_25548_28127.png PT8-14_28128_28439.png PT8-14_28440_31163.png PT8-14_31164_32231.png PT8-14_32232_32879.png PT8-14_32880_34163.png PT8-14_34164_34262.png
Better. Okay. Hi. My name is Jachim Cepitski. I welcome to my talk about how to get data through wireless or even wireless. I have been involved longer than 10 years with development of open source software for geospatial, especially. I started on the desktop side using grad or developing for and with grads GIS. Then I slowly moved to server side and last years, like a couple of years I make my living, I do make my living through development of JavaScript based web, GIS is web applications, whatever. What is this? I have an error here. Okay. You probably know that the development of web based applications has completely changed during last couple of years. We are no longer writing JavaScript code and sending it somehow directly to the client side, like we did in the early days. This is nice. Okay. When I set up the presentation, I check the first slides. They seem to be working. And now, as you may see, there is like nothing. I'm sorry. No, I didn't copy it to the hard drive. I see. Thank you. Okay. Thank you for the link. Nice. Okay. Sorry. Another start. I will have to move faster. Okay. Next. Yeah. Another try. Yeah. Cool. Works. Yeah. So we are no longer sending JavaScript files to the browser. Instead, we are compiling the code again. The code is running faster. We transfer not so many data, not so much the code, the program itself isn't that big. You can compress, for example, code of open layers 3 right now from more than a megabyte than several kilobytes for concrete example. The other side, this is what we are facing right now, or at least a team I am working with. The compilation of the whole application takes like several minutes, and if you are deploying it, and if you won't have continuous integration, it takes really a couple of minutes, so to say, or maybe an hour or two a day to get code compiled. Yep. But nowadays, JavaScript renders and interpreters, they do not have a problem with rendering. Really, really big amounts of data which we could see in the presentations during the whole FOS4G, not only FOS4G. And raster images with them is pretty simply, they are coming with compressed form using JPEGs or PNG files. And for rendering a raster image, there is no rasterization needed anymore, actually, because at the end, there finally is somewhere at the end, you have to transfer the data from the vector to raster, of course, in order to be able to display them. The amount of the information in the raster file can be significantly lower if you compare it to vector file. We can say that the amount of information in a raster file is, of course, limited to its resolution and number of colors you are able to somehow to transfer to the user, so the user can really still interpret it. But the amount of information in a vector file is given by, so to say, number of vertexes, you need for description of a feature, like for example, adoc. And, of course, the number of attributes you are able to put into your vector data. From certain point of, from certain level of detail, the size of a vector data might be bigger, actually, than the size of raster data, though. Why would you need the scope of this presentation is how to transfer vector data to the client? And why would you need this anyway, right? Anyhow. You want the user to enable it, interact with the data. There are dynamic maps. You want to somehow distribute the dynamic aspect of the data. Of course, you want to enable the user to edit the data and more other reasons. But the main reason why I think it's the most important one is that vector data are simply cool. And it gives you all the power of having the data on client side. You can do whatever you need to do with them. But we are facing here the common database parallel. You cannot have all three attributes of the data at once. Even if you keep trying, if you keep pushing, whatever you try to do, you always have to struggle. Either you will have big data or you will deliver the data in a fast way, or the data will be up to date. You can always just do two of those attributes, but never all three of them, whatever you try to do. Those would be the suggested steps when you deal with large amount of data. Of course, of vector data. Various strategies, so to say. It starts from the beginning, from the very simplest one, and then it goes through the most complicated one. The first step is compress the communication between the server and the client. Then the other, choose a proper file format, actually, for the data transfer itself. Use cache, obviously. Use lossy compression for vector data as well. You can do this with rosters, why not for vectors? Transfer only the data, the user really, really needs to see, or the application really, really needs to have. To behave in some way that the user has a feeling of having everything at hand. The first step, compress server-client communication. It might be obvious, right? But you would be probably surprised how often the system administrators forget the simplest thing. Switch on, for example, mod deflate on Apache server. If you take an 80 megabytes big geogson file, it contains 50,000 polygons based on OpenStreetPan data, you can really make it 10 times smaller. Just like that, just with single configuration option. So don't forget this. This is probably the first start. If something is slow, you have a feeling that there is something slow. This would be probably the first check I would do. The other check would be, do I use the proper file format? Everybody is talking about geogson, right, nowadays. But surprisingly, to me, it was a really big surprise. Geogson isn't the most compressed format for vector data. If you compare it to GML, surprisingly, it's twice that big. Again, I'm speaking about 50,000 of polygons based on OpenStreetPan data, randomly picked from the dataset. I don't know actually why this is possible. I have rechecked several times. I have a feeling that the GML file doesn't contain attributes where there is no value, while geogson had really, really all the attributes, even though they were empty, I think so. I just used standard OGR library for data transformation. But on the other hand, of course, file size isn't everything. You can imagine parsing 40 megabytes GML in the browser, takes a certain amount of energy and processor time compared to geogson as well. What I want to say, just be aware of that. Use cache, another strategy. The cache doesn't necessarily have to be, you don't have to set up the whole tile cache stack. Might be enough if you turn on, for example, net framework cache, if you are using net framework for your coding or whatever. And you simply cache just a little amount of your data on the application on the server side, so that the application doesn't have to really touch the database. If nothing has changed, there is no reason for touching the database. And if you want to deliver the data in a fast way, and for some reason, might be data size, might be badly optimization, or badly indexed files in your post-GIS database, whatever. If you turn on cache on your application server, it makes it work really, really much faster than if you go to the data directly. And again, it doesn't cost you basically anything, it's just a few lines of code. And you can have a fast result, actually, very easily. Okay, another strategy. If you are displaying compressed JPEG files, for example, and for users, and everybody knows that it looks nearly as the reality, but users don't complain, it's good enough for them. So why couldn't we do the same with vectors? I was already mentioned here, as was for G as well. Who needs actually 1000 of vertexes per line for, of course, certain zoom level? When, for example, 3 vertexes or 4 would be enough for the data line, or polygon, or whatever representation. Backdoor mapping libraries, the modern ones, like OpenLayers, or OpenLayers 3, of course, or Leaflet, they do actually have a pre-processing of the data, which are at hand, before they render it in, for example, canvas render. So they are really then the most expensive part, the rendering itself. It's using, it's working with really a compressed number of vertexes, if you are rendering lines or polygons. But, of course, that doesn't solve the problem. It's not that you need the data to be transferred to the client. This is everything happening just on the client side, based on the whole dataset. But I would advise you to have a look at the application over there, where you can really play set number of vertexes. You need to perline and see what's happening there. And this is what's happening on the client side, somehow, in the background. One of the formats which actually implements the compression is TopoJSON format. It originates on JoeJSON, and again, my concrete example, if I take 80 megabytes JoeJSON polygons, I convert it to TopoJSON. It has only 3 megabytes, and if I zip it using Modiflate, then it has only 50 kilobytes. It seemed to be magic to me. I would like somebody in the audience to reproduce such experiments, because it seems impossible, but if you render the dataset, it really looks like it's supposed to be. And, yeah, with just a little bit of data processing, you can get pretty good results. TopoJSON was introduced by Bostock. Everybody knows TopoJSON. I think it doesn't make sense to talk much further about it, for one reason that I don't really understand what's behind it, what's behind the magic, except for, yeah, it uses relative coordinates, delta values from some starting point, so to say. And there is an internal index of arcs, so-called and so on. There is a link to documentation on GitHub where you can get into the detail. Sorry, you wanted to make a picture? Okay, the presentation will be online later. Okay, and as I said, it uses less compression, since the delta values are integer values, basically. So you really don't, you lose a certain level of detail, but for most of the applications, it's far enough. But there's one issue I was facing with TopoJSON. There is no support for so-called big integer numbers, which again shouldn't be a problem for 99% of your use cases. But my use case is I want to display a catastrophic parcel data in the web browser, and the RD is big integer then. Of course, there's a workaround, you know, you can create your own IDs and use string, for example, for this one, but yeah, it isn't that nice. And one of the next steps would be you can, you don't have to transfer it, or you don't want to transfer all the data, which are at hand, in one step from the server to the client. So till now we were speaking about data shrinking, but now another strategy, so that the user has a feeling that the application is somehow living, that the data are there, even though they might be coming in several steps. You don't have to transfer everything in one step. We are used to tiled restores, and everybody is speaking about tiled vectors. There were several talks already about tiled vectors. This concrete example is based on openlers, and it uses actually JSON tiles. Okay, another five minutes. It's relatively complicated to get them rendered compared to restores, of course, but nowadays libraries, they support it. It's basically no problem, so what you can see here in the presentation, oh, I can. Great. Is this border between those tiles, but this was a bucket basically in openers and it's gone, so there's no problem with that. Then you need to set up the whole infrastructure that makes things a little bit complicated. One of the possible solutions would be tiles-tiles-project and Python-based project, which generates you, which is able not only to generate raster tiles, but also geo-Json tiles and even topo-Json tiles, which is very nice. I was facing some difficulties using different projection than spherical mercators and Google Mercator or Web Mercator projection. But that was the only geo-ahead with that, otherwise it works nicely. Another standard is mapbox-tiles. Again, we were addressing this during the whole FOS4G. Mapbox has developed and opened a standard for basically binary vector encoding. The first thing is that the coordinates of the data do not use the geographical coordinates anymore, it uses relative coordinates of the image or of the whole map. So rendering calculations, there's no calculations, it renders pretty fast. And of course it's supported in leaflet. In openers, it's going to be supported soon. Soon means that I already tested it on my computer, it's in a branch and it will be there soon. And for generation, you don't have to use mapbox infrastructure, you can use geo-server or again in a very near future map server. I saw it. There might be another strategy you might take. If you have a look at this polygon, apparently it is split in this case and to four different vector features. And it is really like it is in your map application, you will have at least four different features with the same ID, for example. Then if you provide some filtering, some selections, of course the application has to be adjusted to that fact. Another option would be that you download just the vector data, which for example, centroids fit into particle tile. Then it could look like this. You see there are no clear boundaries because I'm downloading whole vector features, which centroid fit into the virtual. So to say with two running box of the tile. Of course, if you have some long power source, long polygon data, for example, or long point data on the edge of the tile, then the vector feature won't be loaded. It depends. But for most of these cases, this seems to be working pretty well. And then you have only one feature, one times as one object in your application. Yes, and result of this effort around it. And user will see just what he or she is expecting when you are dealing with raster tiles. That means there will be somehow loading tiles. The map will slowly appear and the user will have a feeling that something is happening, which is always good. Conclusion. The first question would be, do I need to use all this compression file format, tiling, stack, when I want to display single GPS log file? Answer is obvious. Those approaches are just for situation when the application seems to be slow for some reason. If you are trying to load some bigger amount of data and you have to define the bottlenecks and, yeah, those would be the strategies you can take if you want to reduce the time, the download time, or the rendering time, whatever. Caching and tiling of vector data gives you more speed, but of course you always can have only two attributes out of three. And with caching, you lose the timeliness of the data, because there will always be some cache. For some kind of application, this is perfectly okay. Everybody can live with Google data even though everybody knows they are pre-cached. And you never get the same result asking from different side of the world, asking Google. Or, yeah, you never get the same result on two parts of the world at the same time, because the databases isn't synchronized and so on. They are not up to date. But it's fine. Good enough. Thank you for your attention. Any questions? Yeah, as I said, we have like five minutes, two minutes for questions. So one or two. Okay. Yeah, please. Okay, you need the mic. The mic is running. So you did try to plug OpenLayers to Mapserver? Yes, I did. Yes, it works. We did it just yesterday together with Thomas Monfort and Andrea Sihokover, one from the Mapserver side, the other one from the OpenLayers side, with myData. MyData means really reliving catastrophe data, partial data, and in PostGIS database. I wanted to show it, but I wasn't able to plug my computer into this VGA output. So I can show it to you later, separate and private. Okay, thank you. Then I would like to ask the next speaker.
welcome to my talk about how to get data through JavaScript renders and interpreters, they do not amounts of data which we could see in the course, in order to be able to display them. The you are able to put into your vector data. From are facing here the common database parallel. You cannot have all three attributes of the data at to struggle. Either you will have big data or you will deliver the data in a fast way, or the data suggested steps when you deal with large amount compression for vector data as well. You can do nowadays. But surprisingly, to me, it was a amount of your data on the application on the server side, so that the application doesn't have OpenLayers 3, of course, or Leaflet, they do problem. It's not that you need the data to be look at the application over there, where you can in the background. One of the formats which polygons, I convert it to TopoJSON. It has only 3 and use string, for example, for this one, but the data, which are at hand, in one step from the tiled vectors. This concrete example is based on a little bit complicated. One of the possible solutions would be tiles-tiles-project and the coordinates of the data do not use the that you download just the vector data, which for vector features, which centroid fit into the load some bigger amount of data and you have to the strategies you can take if you want to reduce the time, the download time, or the rendering gives you more speed, but of course you always