Caching time changes in maps
Montanari, Marco
Open History Map aims to display the changes that happened in the past both from a political as well as a social and topographical point of view. the storage of these multi-dimensional changes has an enormous impact on the way the map is visualized. For this reason we needed to develop a caching system that was on one side flexible enough to display a non-quantizable dimension (time) and give us the possibility to pre-cache the whole system into a distributable package for easy local usage and possibly fast update of the package. The caching system itself is backend independent and defines a process to simplify access to a mix of discreet (x,y,z) and continuous (t) identifiers for independently variating geospatial datasets.
 You done<|cy|><|translate|> and Hi. Hi Marco, how are you? Fine, thanks. Are you in Italy right? Yes, I am in Italy. So in the evening after lunch? Yeah, it's half past three. So it's after lunch, but enough after lunch to be already after the nap time. Yeah. So I think we'll also mention Open History Map, right? Yeah, exactly. Oh, good, good. And I think everything is ready. You can upload your presentation. Yeah, I can share about uploading. Okay. Oh, it's just sharing. In fact, it's just sharing. If you click there and share, it will appear here. Perfect. So your second monitor is on top of the other? Yeah, exactly. It's a classic. It looks like I'm looking at answers from gods above, but it's not. I don't know you are near the pope and hearing the voice of... Where are you from, Marco? Bologna, Italy. Bologna? Yeah. Yeah. Good, good, good. Nice city. Okay, let me share my screen here. Have you been here in Bologna? Yes, yes, I've already been there. Oh, great. It's quite rare because usually people just go to Rome, Milan and stuff. Everything is good. Venice, yeah. And Venice, yeah. But those cities in the north is quite rich and the cities are really, really nice. But Bologna has a very much history and so on. Yeah, absolutely. One reason is the agreements of Bologna are very well known in all universities in Europe. So Bologna is... Yeah, that's true. Yeah, we have your presentation here. I can put it on the stage. Perfect. Oh, open history map is there. Yes. Good, good. And we have everything ready. So in a few seconds I'll give you the stage and you can introduce yourself and make your presentation. Yeah. Marco, thank you for being here and in time and everything ready. So the stage is yours. Have a nice presentation. Thank you very much. So hello everyone and... Sorry, I don't see the screen anymore. Okay, hello everyone and welcome to my third presentation this year on catching time changes in maps on open history map. Let me give you a very fast intro to open history map per se. Open history map is an organization we're trying to create tool chains and tools for digital humanities basically to represent data of the past, about the past with modern tools. So to do that, sorry, to do that we have realized, we have generated a system, an infrastructure and a visualization. And so we have several elements to show. So that's the places where cities are, city details and also countries level details in various kinds of scales. So this creates a huge amount of data. It's several gigabytes of data that have to be pushed to the end user. Based on the levels of detail the user is looking at and the area he is looking at. So this is quite complicated to do. And for this reason, we started looking at... For this reason we had to develop an architecture that gives us the ability to do several things automatically. The whole architecture I discussed this yesterday. Today I'm looking just at the part where we look at the data itself, which is this area and it's composed of a front end where the map is displayed with Mapbox GL and the tile server that does just the heavy lifting of showing off the tiles. The service that enables the users to and the importers more than the users to import the data into the database that is a post-JIS database. And then we have the caching system. To explain you the caching system and how we developed that, I have to show you the details about the database. Specifically the database is based on, as I said, post-JIS and very reliant on the partitioning system offered by a post-JIS. Precisely the information is collected with several sets of information connected to the single dataset. It's not based on the same infrastructure and the same idea of OpenStreetMap. Specifically we have two tables, one of the items that are stored in the system, which have two data, two information that represent the moment and the two moment from when and to when this item is relevant on the map. And this fact represents, is represented by two floating point numbers. It's not a time, it is not a data time object, it's not a timestamp, because post-JIS introduced negative timestamps just a few days ago. This is post-JIS 14. One relevant element is the layer and then we have the properties, that is a JSONB object that represents all the details that we want to use for front-end visualization and also for data management. Then we have the author and the geometry is not stored within the item, it's stored within the hash, it's stored as a foreign key created by this hash element that represents the original hash of our geometry. And then within the table geometries we have several copies of the same geometry with various levels of zoom, so that we can recreate the data we're looking at based on the zoom level and the moment in, based on zoom level, moment in time x and y. And x and y are, obviously, zoom level x and y give us the map vector tile and the date represents, it gives us the specific item to look at. As I said, we rely very much on partitioning, meaning that based on the structure of time or the end of production of data over time, we have a, for example, we have a very sparse bar past, meaning that we have one only partition, for example, from 8000 BC to 4000 BC. Then we have a smaller partition from 4000 to 3000, then 3000, 2000, 2500 BC, 1500, 1000, and then we go even in smaller steps up to almost 50-year steps within the last, no, almost five-year steps within the last 50 years, and 50-year steps within the last 200. Because obviously the amount of data we collect and we have managed to collect and use is absolutely distributed in a very irregular way. And this means also that, for example, for geometries we can rely on the fact that all points are points, so we can zoom around however we want, but points are points. On the other hand, the various zoom levels, we have them as well partitioned into several partitions. This results in a very huge amount of partitions, but through SQL alchemy and an automated partitioning script we can generate these partitions automatically based on the layers and the configurations of the zoom levels. This configuration is reused in many parts of the whole infrastructure. This creates for us, for example, a huge amount of data stored in our database based on the various levels, because as I said, the geometries are repeated for the various levels of detail, so it's normal for us to have almost the same size of the tables for the high-level zoom levels and the low-level zoom levels. And this creates obviously a problem with uploads, because it takes time to generate the data. Now this creates also a – anyway, our infrastructure is based on the four volunteers that are working on the whole system, so it's based on not much – let's say it's not paid by huge funding, but in fact it's not paid by funding at all, so it's based on a volunteer-based server. So the architecture is – we have several problems when visualizing the data. By the way, the map is available here. If you want to share the link, it's map.openhistorymap.org, where you can find our details about our project. Anyway, this creates a problem, because when you look at specific moments in time, Europe in the 17th, 18th century is a huge mess, is a horrible mess, is a beautiful mess, honestly, but anyway it's a mess, because there's a lot of data, a lot of changes, a lot of things going on, and this takes up to a minute to download, and the browser ignores the downloaded data, the scripts fail, and everything slows down, and the users are unhappy, and we are unhappy, because our users are unhappy. So why not create a cache, a caching system? So let's go to the drawing board. How do we call our layers? We basically call our layers via a ZXY PBF query. Only we added the date element in the middle, but the date is a float, and this creates a whole lot of problems when trying to cache all these elements, because this is a list, and that's not a problem. This is integer elements, and we know that very well we can cache them any way we want, but this is a floating point, and this makes things horrible. So how could we manage that? And we had this horrible problem that we have a layer, we have the X and the Y, and the elements could be potentially continuous, but looking at it with perspective, we understood that in fact there was a slightly different issue. We had several items that had an OHM from, and an OHM2 for the same item, and an OHM from, and an OHM2. So we had within a potentially continuous dimension, we had very specific moments in time when this dimension changed, and it was possible to preview these changes in our structure. So what happened then? We asked ourselves, why don't we just consider these elements simply to be relevant or irrelevant? And once we look at it this way, this means that basically the map itself changes only in these instance, which means that we just need to store these instance and recreate a new way to manage this transformation. And so we tried just to look at it in this way by saying that we have a transport, for example this is the key, and this is the key, and this is the value. We have two caches basically, we have two key value stores. One key value store that tells us that we need to get the relevant dates for this specific XY set for this layer, and it gives us back a list of valid dates, of a valid sequence of dates. Within that valid sequence of dates, we choose the one that is just before the last relevant change before the date we're looking at. So that this query becomes this specific MBT tile, and then we can return this style to our user. Does this create repetition? Absolutely, but it's part of the caching game. Does this, is this efficient? No, it's just the first draft of our idea of a caching system for time-space information. On the other hand, we had this beautiful thing, we love partitioning, and so we're trying to use partitioning as well for this infrastructure. And so we developed OHM cache. It's based on FastAPI in Python. It's based on level DB. Is this enough for partitioning? Somehow it is, and we think, but that means that we have to do partitioning through load balancing and routing. So it becomes more of a system and operations matter to manage the caching of the whole infrastructure. And we're trying to think about something even more fun with the people from the University of Bologna, which is using Cademlia. It was, I'm old, and so I remember when we used Cademlia for downloading music and movies when internet connections were not that fast. But the idea is to go to try at least to see how it works with a distributed hash tape. The code is not yet uploaded. I was trying to upload it this morning, but some things came up, and I had no time to do that. But I will be uploading it in a few minutes after the presentation, and it will be in GitHub, open history map, OHM cache. And well, thanks. We have questions, details, anything else. Very fast presentation for the Friday afternoon. Your mic is off. Yes, thank you. Thank you, Marco. And thank you for your quick presentation, although I think it was clear. And it is complicated for people on the humanistic side. It is not easy to explain all this stuff, I think, to them. You cannot show the behind the scenes. No, it is absolutely not. It is not easy, but it's always fascinating to see them come around and understanding side effects of getting the good part of the concepts behind these tools. For example, as I explained yesterday, and as you might have seen in the first slides, there is a lot of data about the past. And let me just share with you another image, just because we have still 15 minutes. Let me show you a few additional elements on the project. Here, let me change the screen. If you can show the screen I'm sharing right now. You can see this is not yet connected to the cache. And you see the central Europe is very slow to download, because obviously the changes in time have been so enormous and the data to be downloaded is quite complicated. Because on the other hand, we are trying to keep track of several kinds of data sets. For example, while having almost a lot of data from a national level, national size data sets, we also have, for example, for Bologna, where I am, the city structure over time. And we can see how it changed from the 14th century to the 17th and to the late 18th century, where we also have almost all buildings that were in and around the city at the time. It takes a few seconds to download. But for many buildings, we also have additional data as well, meaning that we can also go exactly. We also have data about the height of many buildings. We had the number of levels that are there in a specific building. For some of them, it's obviously just reconstructed from archivistic information. For some of them, the buildings are still there. So you see the amount of data we have to jiggle with is quite a lot. So a caching system becomes every day more important. And for this reason, we started working on this. We started discussing this with small tests. It works, but it's not stable enough to have it yet up and running. But we will try to have it running by the end of October. And as always, and as every other tool created within this project, it's always going to be open sourced and available for anyone to work on, play with, and reuse for any other kind of usage. So Marco, while you are showing this, and thank you for showing this example, this detailed example from Polonia. Are the towers still upright in the 19th century? Yeah, sadly, the hate of the towers is not within the data set we use as a baseline. Yeah, it's too bad. Last, in Dar, two years ago, we presented a project where we were using the 3D models of the real towers as elements here. But the data from that project is not yet integrated into this new version of the interface. So as always, everything is working in progress. OK, so the first question, the most voted question, is about partitioning postage. And how do you configure this kind of partitioning? Yes, let me show you. The partitioning in post-GIS is something absolutely beautiful. And based on the, we're using a set of, how do we call it? Let me, I'm looking for the files here. OK, here. Basically, I'm generating the models, the data structure of the partitioning based on a script that is also open source, or if it is not, it will be when we release all the data. And basically, it tries to generate a partition for every structure within the labels, for every label, for every layer that we have to use, and for every structure we have to manage. For all of the time partitions we are using. So as you can see, generic minus infinite minus to 4,000, minus 3,500, and so on. And so with this partitioning system, we are able to generate something like, it's 15 layers by around 50 time blocks. So it's a huge amount of layers to be generated, to be a huge amount of partitions to be generated. And it's all done with SQL Alchemy. OK, OK, on the Python side, right? Yes, exactly. There is one question about timescale DB plugin. So the question is, have you looked at the timescale DB plugin? Not enough, honestly, not enough. Because the answer to that is yes and no. Because we will be using timescale DB for a part of the project that is based on real-time serious data. The problem with this is that it relies a lot on the fact that the problem is exactly the fact that we are not using it. We were trying to use it with time buckets for geographic information. But it was somehow inefficient on our side considering that we have elements that have to go beyond the simple. We have potentially objects that go through several time buckets, and so it was quite problematic. But we will try to use it again, because, again, as I said, we will be using it for sure for a side project that is already ongoing with the University of Bologna that has the aim to generate something like a data warehouse for indexes, for several kinds of objects of the past, normalized through Wikipedia, and with data IDs so that we can look at how was the GDP of ancient Egypt and so on. We have also another question, and it's about the open historical map, which is a similar name, but is this project similar? No, it's similar. We have almost the same object, and we had a very interesting call in a few last month, because the object is similar, but we have a different approach. Well, open history map, open historical map relies on, yeah, it's a mess, because we almost started in the same period. So we both started and we both grew a little bit towards the larger data sets, and now it's too late to change it. And so we discussed the whole thing and said, okay, it's too late, but as long as we're discussing almost the same thing with different names, who cares? It's data. If you want our data, if we want your data, it's always data. So anyway, the approach is slightly different. We are, with an open history map project, we want to have more control over the academic aspect of the data sets, and so we have the data index that you can reach at index.openhistorymap.org. And with that, we want to try to manage the complexity of the data and the data quality with a separate element. While open historical map tries to do that with the open street map approach, which is slightly different, more community-based, more structure-based. Our approach is more towards information that is verified and controlled by academia or academic procedures. We're not directly importing data. We're importing data for people who generate data. And on the other hand, the other great difference is that we want to track also ephemera information, meaning where was somebody at some point in time? What were the movements of one chip over time? And so with these information, this is our information that open historical map does not have and does not want to have because it's open street map of the past. And for that reason, we will most probably in the next month try to integrate that data from our database, from our interface, from our system into open historical map so that we can have a shared structure for event visualization and something like open... Let's say we also have something like historical street view where we look at historical photos and paintings and show them within their position in time and space. So that creates a whole set of information that, again, open historical map does not have and does not want to have while we try to use it for digital humanities and so becoming a more structured approach. Yeah, thank you, Mark, for this explanation about both projects. And we will end here this session. Thank you, Mark, for the presenters for being here and sharing their knowledge. We have the OSM General Meeting in a few minutes and you are invited to participate in the life of the association and the board is there and please come to the general meeting of OSGEO. Thank you all and see you around.
 You done  and  Hi.  Hi Marco, how are you?  Fine, thanks.  Are you in Italy right?  Yes, I am in Italy.  So in the evening after lunch?  Yeah, it's half past three.  So it's after lunch, but enough after lunch to be already after the nap time.  Yeah.  So I think we'll also mention Open History Map, right?  Yeah, exactly.  Oh, good, good.  And I think everything is ready.  You can upload your presentation.  Yeah, I can share about uploading.  Okay.  Oh, it's just sharing.  In fact, it's just sharing.  If you click there and share, it will appear here.  Perfect.  So your second monitor is on top of the other?  Yeah, exactly.  It's a classic.  It looks like I'm looking at answers from gods above, but it's not.  I don't know you are near the pope and hearing the voice of...  Where are you from, Marco?  Bologna, Italy.  Bologna?  Yeah.  Yeah.  Good, good, good.  Nice city.  Okay, let me share my screen here.  Have you been here in Bologna?  Yes, yes, I've already been there.  Oh, great.  It's quite rare because usually people just go to Rome, Milan and stuff.  Everything is good.  Venice, yeah.  And Venice, yeah.  But those cities in the north is quite rich and the cities are really, really nice.  But Bologna has a very much history and so on.  Yeah, absolutely.  One reason is the agreements of Bologna are very well known in all universities in Europe.  So Bologna is...  Yeah, that's true.  Yeah, we have your presentation here.  I can put it on the stage.  Perfect.  Oh, open history map is there.  Yes.  Good, good.  And we have everything ready.  So in a few seconds I'll give you the stage and you can introduce yourself and make your  presentation.  Yeah.  Marco, thank you for being here and in time and everything ready.  So the stage is yours.  Have a nice presentation.  Thank you very much.  So hello everyone and...  Sorry, I don't see the screen anymore.  Okay, hello everyone and welcome to my third presentation this year on catching time changes  in maps on open history map.  Let me give you a very fast intro to open history map per se.  Open history map is an organization we're trying to create tool chains and tools for  digital humanities basically to represent data of the past, about the past with modern  tools.  So to do that, sorry, to do that we have realized, we have generated a system, an infrastructure  and a visualization.  And so we have several elements to show.  So that's the places where cities are, city details and also countries level details in  various kinds of scales.  So this creates a huge amount of data.  It's several gigabytes of data that have to be pushed to the end user.  Based on the levels of detail the user is looking at and the area he is looking at.  So this is quite complicated to do.  And for this reason, we started looking at...  For this reason we had to develop an architecture that gives us the ability to do several things  automatically.  The whole architecture I discussed this yesterday.  Today I'm looking just at the part where we look at the data itself, which is this area  and it's composed of a front end where the map is displayed with Mapbox GL and the tile  server that does just the heavy lifting of showing off the tiles.  The service that enables the users to and the importers more than the users to import  the data into the database that is a post-JIS database.  And then we have the caching system.  To explain you the caching system and how we developed that, I have to show you the details  about the database.  Specifically the database is based on, as I said, post-JIS and very reliant on the partitioning  system offered by a post-JIS.  Precisely the information is collected with several sets of information connected to the  single dataset.  It's not based on the same infrastructure and the same idea of OpenStreetMap.  Specifically we have two tables, one of the items that are stored in the system, which  have two data, two information that represent the moment and the two moment from when and  to when this item is relevant on the map.  And this fact represents, is represented by two floating point numbers.  It's not a time, it is not a data time object, it's not a timestamp, because post-JIS introduced  negative timestamps just a few days ago.  This is post-JIS 14.  One relevant element is the layer and then we have the properties, that is a JSONB object  that represents all the details that we want to use for front-end visualization and also  for data management.  Then we have the author and the geometry is not stored within the item, it's stored within  the hash, it's stored as a foreign key created by this hash element that represents the original  hash of our geometry.  And then within the table geometries we have several copies of the same geometry with various  levels of zoom, so that we can recreate the data we're looking at based on the zoom level  and the moment in, based on zoom level, moment in time x and y.  And x and y are, obviously, zoom level x and y give us the map vector tile and the date  represents, it gives us the specific item to look at.  As I said, we rely very much on partitioning, meaning that based on the structure of time  or the end of production of data over time, we have a, for example, we have a very sparse  bar past, meaning that we have one only partition, for example, from 8000 BC to 4000 BC.  Then we have a smaller partition from 4000 to 3000, then 3000, 2000, 2500 BC, 1500, 1000,  and then we go even in smaller steps up to almost 50-year steps within the last, no,  almost five-year steps within the last 50 years, and 50-year steps within the last 200.  Because obviously the amount of data we collect and we have managed to collect and use is  absolutely distributed in a very irregular way.  And this means also that, for example, for geometries we can rely on the fact that all  points are points, so we can zoom around however we want, but points are points.  On the other hand, the various zoom levels, we have them as well partitioned into several  partitions.  This results in a very huge amount of partitions, but through SQL alchemy and an automated partitioning  script we can generate these partitions automatically based on the layers and the configurations  of the zoom levels.  This configuration is reused in many parts of the whole infrastructure.  This creates for us, for example, a huge amount of data stored in our database based on the  various levels, because as I said, the geometries are repeated for the various levels of detail,  so it's normal for us to have almost the same size of the tables for the high-level zoom  levels and the low-level zoom levels.  And this creates obviously a problem with uploads, because it takes time to generate  the data.  Now this creates also a – anyway, our infrastructure is based on the four volunteers that are working  on the whole system, so it's based on not much – let's say it's not paid by huge  funding, but in fact it's not paid by funding at all, so it's based on a volunteer-based  server.  So the architecture is – we have several problems when visualizing the data.  By the way, the map is available here.  If you want to share the link, it's map.openhistorymap.org, where you can find our details about our  project.  Anyway, this creates a problem, because when you look at specific moments in time, Europe  in the 17th, 18th century is a huge mess, is a horrible mess, is a beautiful mess, honestly,  but anyway it's a mess, because there's a lot of data, a lot of changes, a lot of  things going on, and this takes up to a minute to download, and the browser ignores the downloaded  data, the scripts fail, and everything slows down, and the users are unhappy, and we are  unhappy, because our users are unhappy.  So why not create a cache, a caching system?  So let's go to the drawing board.  How do we call our layers?  We basically call our layers via a ZXY PBF query.  Only we added the date element in the middle, but the date is a float, and this creates  a whole lot of problems when trying to cache all these elements, because this is a list,  and that's not a problem.  This is integer elements, and we know that very well we can cache them any way we want,  but this is a floating point, and this makes things horrible.  So how could we manage that?  And we had this horrible problem that we have a layer, we have the X and the Y, and the  elements could be potentially continuous, but looking at it with perspective, we understood  that in fact there was a slightly different issue.  We had several items that had an OHM from, and an OHM2 for the same item, and an OHM  from, and an OHM2.  So we had within a potentially continuous dimension, we had very specific moments in  time when this dimension changed, and it was possible to preview these changes in our structure.  So what happened then?  We asked ourselves, why don't we just consider these elements simply to be relevant or irrelevant?  And once we look at it this way, this means that basically the map itself changes only  in these instance, which means that we just need to store these instance and recreate  a new way to manage this transformation.  And so we tried just to look at it in this way by saying that we have a transport, for  example this is the key, and this is the key, and this is the value.  We have two caches basically, we have two key value stores.  One key value store that tells us that we need to get the relevant dates for this specific  XY set for this layer, and it gives us back a list of valid dates, of a valid sequence  of dates.  Within that valid sequence of dates, we choose the one that is just before the last relevant  change before the date we're looking at.  So that this query becomes this specific MBT tile, and then we can return this style to  our user.  Does this create repetition?  Absolutely, but it's part of the caching game.  Does this, is this efficient?  No, it's just the first draft of our idea of a caching system for time-space information.  On the other hand, we had this beautiful thing, we love partitioning, and so we're trying  to use partitioning as well for this infrastructure.  And so we developed OHM cache.  It's based on FastAPI in Python.  It's based on level DB.  Is this enough for partitioning?  Somehow it is, and we think, but that means that we have to do partitioning through load  balancing and routing.  So it becomes more of a system and operations matter to manage the caching of the whole  infrastructure.  And we're trying to think about something even more fun with the people from the University  of Bologna, which is using Cademlia.  It was, I'm old, and so I remember when we used Cademlia for downloading music and movies  when internet connections were not that fast.  But the idea is to go to try at least to see how it works with a distributed hash tape.  The code is not yet uploaded.  I was trying to upload it this morning, but some things came up, and I had no time to  do that.  But I will be uploading it in a few minutes after the presentation, and it will be in GitHub,  open history map, OHM cache.  And well, thanks.  We have questions, details, anything else.  Very fast presentation for the Friday afternoon.  Your mic is off.  Yes, thank you.  Thank you, Marco.  And thank you for your quick presentation, although I think it was clear.  And it is complicated for people on the humanistic side.  It is not easy to explain all this stuff, I think, to them.  You cannot show the behind the scenes.  No, it is absolutely not.  It is not easy, but it's always fascinating to see them come around and understanding  side effects of getting the good part of the concepts behind these tools.  For example, as I explained yesterday, and as you might have seen in the first slides,  there is a lot of data about the past.  And let me just share with you another image, just because we have still 15 minutes.  Let me show you a few additional elements on the project.  Here, let me change the screen.  If you can show the screen I'm sharing right now.  You can see this is not yet connected to the cache.  And you see the central Europe is very slow to download, because obviously the changes  in time have been so enormous and the data to be downloaded is quite complicated.  Because on the other hand, we are trying to keep track of several kinds of data sets.  For example, while having almost a lot of data from a national level, national size data  sets, we also have, for example, for Bologna, where I am, the city structure over time.  And we can see how it changed from the 14th century to the 17th and to the late 18th century,  where we also have almost all buildings that were in and around the city at the time.  It takes a few seconds to download.  But for many buildings, we also have additional data as well, meaning that we can also go  exactly.  We also have data about the height of many buildings.  We had the number of levels that are there in a specific building.  For some of them, it's obviously just reconstructed from archivistic information.  For some of them, the buildings are still there.  So you see the amount of data we have to jiggle with is quite a lot.  So a caching system becomes every day more important.  And for this reason, we started working on this.  We started discussing this with small tests.  It works, but it's not stable enough to have it yet up and running.  But we will try to have it running by the end of October.  And as always, and as every other tool created within this project, it's always going to  be open sourced and available for anyone to work on, play with, and reuse for any other  kind of usage.  So Marco, while you are showing this, and thank you for showing this example, this  detailed example from Polonia.  Are the towers still upright in the 19th century?  Yeah, sadly, the hate of the towers is not within the data set we use as a baseline.  Yeah, it's too bad.  Last, in Dar, two years ago, we presented a project where we were using the 3D models  of the real towers as elements here.  But the data from that project is not yet integrated into this new version of the interface.  So as always, everything is working in progress.  OK, so the first question, the most voted question, is about partitioning postage.  And how do you configure this kind of partitioning?  Yes, let me show you.  The partitioning in post-GIS is something absolutely beautiful.  And based on the, we're using a set of, how do we call it?  Let me, I'm looking for the files here.  OK, here.  Basically, I'm generating the models, the data structure of the partitioning based on  a script that is also open source, or if it is not, it will be when we release all the  data.  And basically, it tries to generate a partition for every structure within the labels, for  every label, for every layer that we have to use, and for every structure we have to  manage.  For all of the time partitions we are using.  So as you can see, generic minus infinite minus to 4,000, minus 3,500, and so on.  And so with this partitioning system, we are able to generate something like, it's 15 layers  by around 50 time blocks.  So it's a huge amount of layers to be generated, to be a huge amount of partitions to be generated.  And it's all done with SQL Alchemy.  OK, OK, on the Python side, right?  Yes, exactly.  There is one question about timescale DB plugin.  So the question is, have you looked at the timescale DB plugin?  Not enough, honestly, not enough.  Because the answer to that is yes and no.  Because we will be using timescale DB for a part of the project that is based on real-time  serious data.  The problem with this is that it relies a lot on the fact that the problem is exactly  the fact that we are not using it.  We were trying to use it with time buckets for geographic information.  But it was somehow inefficient on our side considering that we have elements that have  to go beyond the simple.  We have potentially objects that go through several time buckets, and so it was quite  problematic.  But we will try to use it again, because, again, as I said, we will be using it for  sure for a side project that is already ongoing with the University of Bologna that has the  aim to generate something like a data warehouse for indexes, for several kinds of objects  of the past, normalized through Wikipedia, and with data IDs so that we can look at how  was the GDP of ancient Egypt and so on.  We have also another question, and it's about the open historical map, which is a similar  name, but is this project similar?  No, it's similar.  We have almost the same object, and we had a very interesting call in a few last month,  because the object is similar, but we have a different approach.  Well, open history map, open historical map relies on, yeah, it's a mess, because we  almost started in the same period.  So we both started and we both grew a little bit towards the larger data sets, and now  it's too late to change it.  And so we discussed the whole thing and said, okay, it's too late, but as long as we're  discussing almost the same thing with different names, who cares?  It's data.  If you want our data, if we want your data, it's always data.  So anyway, the approach is slightly different.  We are, with an open history map project, we want to have more control over the academic  aspect of the data sets, and so we have the data index that you can reach at index.openhistorymap.org.  And with that, we want to try to manage the complexity of the data and the data quality  with a separate element.  While open historical map tries to do that with the open street map approach, which is  slightly different, more community-based, more structure-based.  Our approach is more towards information that is verified and controlled by academia  or academic procedures.  We're not directly importing data.  We're importing data for people who generate data.  And on the other hand, the other great difference is that we want to track also ephemera information,  meaning where was somebody at some point in time?  What were the movements of one chip over time?  And so with these information, this is our information that open historical map does  not have and does not want to have because it's open street map of the past.  And for that reason, we will most probably in the next month try to integrate that data  from our database, from our interface, from our system into open historical map so that  we can have a shared structure for event visualization and something like open...  Let's say we also have something like historical street view where we look at historical photos  and paintings and show them within their position in time and space.  So that creates a whole set of information that, again, open historical map does not have  and does not want to have while we try to use it for digital humanities and so becoming  a more structured approach.  Yeah, thank you, Mark, for this explanation about both projects.  And we will end here this session.  Thank you, Mark, for the presenters for being here and sharing their knowledge.  We have the OSM General Meeting in a few minutes and you are invited to participate in the  life of the association and the board is there and please come to the general meeting of  OSGEO.  Thank you all and see you around.
FOSS4G_2021_-_Caching_time_changes_in_maps_xhWfnq-pTQo_0_359.png FOSS4G_2021_-_Caching_time_changes_in_maps_xhWfnq-pTQo_360_491.png FOSS4G_2021_-_Caching_time_changes_in_maps_xhWfnq-pTQo_492_551.png FOSS4G_2021_-_Caching_time_changes_in_maps_xhWfnq-pTQo_552_4739.png FOSS4G_2021_-_Caching_time_changes_in_maps_xhWfnq-pTQo_4740_5507.png FOSS4G_2021_-_Caching_time_changes_in_maps_xhWfnq-pTQo_5508_6575.png FOSS4G_2021_-_Caching_time_changes_in_maps_xhWfnq-pTQo_6576_6623.png FOSS4G_2021_-_Caching_time_changes_in_maps_xhWfnq-pTQo_6624_6695.png FOSS4G_2021_-_Caching_time_changes_in_maps_xhWfnq-pTQo_6696_8627.png FOSS4G_2021_-_Caching_time_changes_in_maps_xhWfnq-pTQo_8628_9563.png FOSS4G_2021_-_Caching_time_changes_in_maps_xhWfnq-pTQo_9564_13367.png FOSS4G_2021_-_Caching_time_changes_in_maps_xhWfnq-pTQo_13368_15035.png FOSS4G_2021_-_Caching_time_changes_in_maps_xhWfnq-pTQo_15036_18779.png FOSS4G_2021_-_Caching_time_changes_in_maps_xhWfnq-pTQo_18780_19559.png FOSS4G_2021_-_Caching_time_changes_in_maps_xhWfnq-pTQo_19560_20579.png FOSS4G_2021_-_Caching_time_changes_in_maps_xhWfnq-pTQo_20580_29351.png FOSS4G_2021_-_Caching_time_changes_in_maps_xhWfnq-pTQo_29352_30263.png FOSS4G_2021_-_Caching_time_changes_in_maps_xhWfnq-pTQo_30264_30779.png FOSS4G_2021_-_Caching_time_changes_in_maps_xhWfnq-pTQo_30780_30839.png FOSS4G_2021_-_Caching_time_changes_in_maps_xhWfnq-pTQo_30840_31247.png FOSS4G_2021_-_Caching_time_changes_in_maps_xhWfnq-pTQo_31248_31307.png FOSS4G_2021_-_Caching_time_changes_in_maps_xhWfnq-pTQo_31308_31487.png FOSS4G_2021_-_Caching_time_changes_in_maps_xhWfnq-pTQo_31488_31751.png FOSS4G_2021_-_Caching_time_changes_in_maps_xhWfnq-pTQo_31752_33215.png FOSS4G_2021_-_Caching_time_changes_in_maps_xhWfnq-pTQo_33216_33371.png FOSS4G_2021_-_Caching_time_changes_in_maps_xhWfnq-pTQo_33372_33551.png FOSS4G_2021_-_Caching_time_changes_in_maps_xhWfnq-pTQo_33552_33647.png FOSS4G_2021_-_Caching_time_changes_in_maps_xhWfnq-pTQo_33648_35063.png FOSS4G_2021_-_Caching_time_changes_in_maps_xhWfnq-pTQo_35064_35219.png FOSS4G_2021_-_Caching_time_changes_in_maps_xhWfnq-pTQo_35220_35567.png FOSS4G_2021_-_Caching_time_changes_in_maps_xhWfnq-pTQo_35568_35867.png FOSS4G_2021_-_Caching_time_changes_in_maps_xhWfnq-pTQo_35868_35915.png FOSS4G_2021_-_Caching_time_changes_in_maps_xhWfnq-pTQo_35916_36383.png FOSS4G_2021_-_Caching_time_changes_in_maps_xhWfnq-pTQo_36384_38063.png FOSS4G_2021_-_Caching_time_changes_in_maps_xhWfnq-pTQo_38064_40031.png FOSS4G_2021_-_Caching_time_changes_in_maps_xhWfnq-pTQo_40032_48467.png FOSS4G_2021_-_Caching_time_changes_in_maps_xhWfnq-pTQo_48468_49379.png FOSS4G_2021_-_Caching_time_changes_in_maps_xhWfnq-pTQo_49380_49720.png
You done<|cy|><|translate|> and Hi. Hi Marco, how are you? Fine, thanks. Are you in Italy right? Yes, I am in Italy. So in the evening after lunch? Yeah, it's half past three. So it's after lunch, but enough after lunch to be already after the nap time. Yeah. So I think we'll also mention Open History Map, right? Yeah, exactly. Oh, good, good. And I think everything is ready. You can upload your presentation. Yeah, I can share about uploading. Okay. Oh, it's just sharing. In fact, it's just sharing. If you click there and share, it will appear here. Perfect. So your second monitor is on top of the other? Yeah, exactly. It's a classic. It looks like I'm looking at answers from gods above, but it's not. I don't know you are near the pope and hearing the voice of... Where are you from, Marco? Bologna, Italy. Bologna? Yeah. Yeah. Good, good, good. Nice city. Okay, let me share my screen here. Have you been here in Bologna? Yes, yes, I've already been there. Oh, great. It's quite rare because usually people just go to Rome, Milan and stuff. Everything is good. Venice, yeah. And Venice, yeah. But those cities in the north is quite rich and the cities are really, really nice. But Bologna has a very much history and so on. Yeah, absolutely. One reason is the agreements of Bologna are very well known in all universities in Europe. So Bologna is... Yeah, that's true. Yeah, we have your presentation here. I can put it on the stage. Perfect. Oh, open history map is there. Yes. Good, good. And we have everything ready. So in a few seconds I'll give you the stage and you can introduce yourself and make your presentation. Yeah. Marco, thank you for being here and in time and everything ready. So the stage is yours. Have a nice presentation. Thank you very much. So hello everyone and... Sorry, I don't see the screen anymore. Okay, hello everyone and welcome to my third presentation this year on catching time changes in maps on open history map. Let me give you a very fast intro to open history map per se. Open history map is an organization we're trying to create tool chains and tools for digital humanities basically to represent data of the past, about the past with modern tools. So to do that, sorry, to do that we have realized, we have generated a system, an infrastructure and a visualization. And so we have several elements to show. So that's the places where cities are, city details and also countries level details in various kinds of scales. So this creates a huge amount of data. It's several gigabytes of data that have to be pushed to the end user. Based on the levels of detail the user is looking at and the area he is looking at. So this is quite complicated to do. And for this reason, we started looking at... For this reason we had to develop an architecture that gives us the ability to do several things automatically. The whole architecture I discussed this yesterday. Today I'm looking just at the part where we look at the data itself, which is this area and it's composed of a front end where the map is displayed with Mapbox GL and the tile server that does just the heavy lifting of showing off the tiles. The service that enables the users to and the importers more than the users to import the data into the database that is a post-JIS database. And then we have the caching system. To explain you the caching system and how we developed that, I have to show you the details about the database. Specifically the database is based on, as I said, post-JIS and very reliant on the partitioning system offered by a post-JIS. Precisely the information is collected with several sets of information connected to the single dataset. It's not based on the same infrastructure and the same idea of OpenStreetMap. Specifically we have two tables, one of the items that are stored in the system, which have two data, two information that represent the moment and the two moment from when and to when this item is relevant on the map. And this fact represents, is represented by two floating point numbers. It's not a time, it is not a data time object, it's not a timestamp, because post-JIS introduced negative timestamps just a few days ago. This is post-JIS 14. One relevant element is the layer and then we have the properties, that is a JSONB object that represents all the details that we want to use for front-end visualization and also for data management. Then we have the author and the geometry is not stored within the item, it's stored within the hash, it's stored as a foreign key created by this hash element that represents the original hash of our geometry. And then within the table geometries we have several copies of the same geometry with various levels of zoom, so that we can recreate the data we're looking at based on the zoom level and the moment in, based on zoom level, moment in time x and y. And x and y are, obviously, zoom level x and y give us the map vector tile and the date represents, it gives us the specific item to look at. As I said, we rely very much on partitioning, meaning that based on the structure of time or the end of production of data over time, we have a, for example, we have a very sparse bar past, meaning that we have one only partition, for example, from 8000 BC to 4000 BC. Then we have a smaller partition from 4000 to 3000, then 3000, 2000, 2500 BC, 1500, 1000, and then we go even in smaller steps up to almost 50-year steps within the last, no, almost five-year steps within the last 50 years, and 50-year steps within the last 200. Because obviously the amount of data we collect and we have managed to collect and use is absolutely distributed in a very irregular way. And this means also that, for example, for geometries we can rely on the fact that all points are points, so we can zoom around however we want, but points are points. On the other hand, the various zoom levels, we have them as well partitioned into several partitions. This results in a very huge amount of partitions, but through SQL alchemy and an automated partitioning script we can generate these partitions automatically based on the layers and the configurations of the zoom levels. This configuration is reused in many parts of the whole infrastructure. This creates for us, for example, a huge amount of data stored in our database based on the various levels, because as I said, the geometries are repeated for the various levels of detail, so it's normal for us to have almost the same size of the tables for the high-level zoom levels and the low-level zoom levels. And this creates obviously a problem with uploads, because it takes time to generate the data. Now this creates also a – anyway, our infrastructure is based on the four volunteers that are working on the whole system, so it's based on not much – let's say it's not paid by huge funding, but in fact it's not paid by funding at all, so it's based on a volunteer-based server. So the architecture is – we have several problems when visualizing the data. By the way, the map is available here. If you want to share the link, it's map.openhistorymap.org, where you can find our details about our project. Anyway, this creates a problem, because when you look at specific moments in time, Europe in the 17th, 18th century is a huge mess, is a horrible mess, is a beautiful mess, honestly, but anyway it's a mess, because there's a lot of data, a lot of changes, a lot of things going on, and this takes up to a minute to download, and the browser ignores the downloaded data, the scripts fail, and everything slows down, and the users are unhappy, and we are unhappy, because our users are unhappy. So why not create a cache, a caching system? So let's go to the drawing board. How do we call our layers? We basically call our layers via a ZXY PBF query. Only we added the date element in the middle, but the date is a float, and this creates a whole lot of problems when trying to cache all these elements, because this is a list, and that's not a problem. This is integer elements, and we know that very well we can cache them any way we want, but this is a floating point, and this makes things horrible. So how could we manage that? And we had this horrible problem that we have a layer, we have the X and the Y, and the elements could be potentially continuous, but looking at it with perspective, we understood that in fact there was a slightly different issue. We had several items that had an OHM from, and an OHM2 for the same item, and an OHM from, and an OHM2. So we had within a potentially continuous dimension, we had very specific moments in time when this dimension changed, and it was possible to preview these changes in our structure. So what happened then? We asked ourselves, why don't we just consider these elements simply to be relevant or irrelevant? And once we look at it this way, this means that basically the map itself changes only in these instance, which means that we just need to store these instance and recreate a new way to manage this transformation. And so we tried just to look at it in this way by saying that we have a transport, for example this is the key, and this is the key, and this is the value. We have two caches basically, we have two key value stores. One key value store that tells us that we need to get the relevant dates for this specific XY set for this layer, and it gives us back a list of valid dates, of a valid sequence of dates. Within that valid sequence of dates, we choose the one that is just before the last relevant change before the date we're looking at. So that this query becomes this specific MBT tile, and then we can return this style to our user. Does this create repetition? Absolutely, but it's part of the caching game. Does this, is this efficient? No, it's just the first draft of our idea of a caching system for time-space information. On the other hand, we had this beautiful thing, we love partitioning, and so we're trying to use partitioning as well for this infrastructure. And so we developed OHM cache. It's based on FastAPI in Python. It's based on level DB. Is this enough for partitioning? Somehow it is, and we think, but that means that we have to do partitioning through load balancing and routing. So it becomes more of a system and operations matter to manage the caching of the whole infrastructure. And we're trying to think about something even more fun with the people from the University of Bologna, which is using Cademlia. It was, I'm old, and so I remember when we used Cademlia for downloading music and movies when internet connections were not that fast. But the idea is to go to try at least to see how it works with a distributed hash tape. The code is not yet uploaded. I was trying to upload it this morning, but some things came up, and I had no time to do that. But I will be uploading it in a few minutes after the presentation, and it will be in GitHub, open history map, OHM cache. And well, thanks. We have questions, details, anything else. Very fast presentation for the Friday afternoon. Your mic is off. Yes, thank you. Thank you, Marco. And thank you for your quick presentation, although I think it was clear. And it is complicated for people on the humanistic side. It is not easy to explain all this stuff, I think, to them. You cannot show the behind the scenes. No, it is absolutely not. It is not easy, but it's always fascinating to see them come around and understanding side effects of getting the good part of the concepts behind these tools. For example, as I explained yesterday, and as you might have seen in the first slides, there is a lot of data about the past. And let me just share with you another image, just because we have still 15 minutes. Let me show you a few additional elements on the project. Here, let me change the screen. If you can show the screen I'm sharing right now. You can see this is not yet connected to the cache. And you see the central Europe is very slow to download, because obviously the changes in time have been so enormous and the data to be downloaded is quite complicated. Because on the other hand, we are trying to keep track of several kinds of data sets. For example, while having almost a lot of data from a national level, national size data sets, we also have, for example, for Bologna, where I am, the city structure over time. And we can see how it changed from the 14th century to the 17th and to the late 18th century, where we also have almost all buildings that were in and around the city at the time. It takes a few seconds to download. But for many buildings, we also have additional data as well, meaning that we can also go exactly. We also have data about the height of many buildings. We had the number of levels that are there in a specific building. For some of them, it's obviously just reconstructed from archivistic information. For some of them, the buildings are still there. So you see the amount of data we have to jiggle with is quite a lot. So a caching system becomes every day more important. And for this reason, we started working on this. We started discussing this with small tests. It works, but it's not stable enough to have it yet up and running. But we will try to have it running by the end of October. And as always, and as every other tool created within this project, it's always going to be open sourced and available for anyone to work on, play with, and reuse for any other kind of usage. So Marco, while you are showing this, and thank you for showing this example, this detailed example from Polonia. Are the towers still upright in the 19th century? Yeah, sadly, the hate of the towers is not within the data set we use as a baseline. Yeah, it's too bad. Last, in Dar, two years ago, we presented a project where we were using the 3D models of the real towers as elements here. But the data from that project is not yet integrated into this new version of the interface. So as always, everything is working in progress. OK, so the first question, the most voted question, is about partitioning postage. And how do you configure this kind of partitioning? Yes, let me show you. The partitioning in post-GIS is something absolutely beautiful. And based on the, we're using a set of, how do we call it? Let me, I'm looking for the files here. OK, here. Basically, I'm generating the models, the data structure of the partitioning based on a script that is also open source, or if it is not, it will be when we release all the data. And basically, it tries to generate a partition for every structure within the labels, for every label, for every layer that we have to use, and for every structure we have to manage. For all of the time partitions we are using. So as you can see, generic minus infinite minus to 4,000, minus 3,500, and so on. And so with this partitioning system, we are able to generate something like, it's 15 layers by around 50 time blocks. So it's a huge amount of layers to be generated, to be a huge amount of partitions to be generated. And it's all done with SQL Alchemy. OK, OK, on the Python side, right? Yes, exactly. There is one question about timescale DB plugin. So the question is, have you looked at the timescale DB plugin? Not enough, honestly, not enough. Because the answer to that is yes and no. Because we will be using timescale DB for a part of the project that is based on real-time serious data. The problem with this is that it relies a lot on the fact that the problem is exactly the fact that we are not using it. We were trying to use it with time buckets for geographic information. But it was somehow inefficient on our side considering that we have elements that have to go beyond the simple. We have potentially objects that go through several time buckets, and so it was quite problematic. But we will try to use it again, because, again, as I said, we will be using it for sure for a side project that is already ongoing with the University of Bologna that has the aim to generate something like a data warehouse for indexes, for several kinds of objects of the past, normalized through Wikipedia, and with data IDs so that we can look at how was the GDP of ancient Egypt and so on. We have also another question, and it's about the open historical map, which is a similar name, but is this project similar? No, it's similar. We have almost the same object, and we had a very interesting call in a few last month, because the object is similar, but we have a different approach. Well, open history map, open historical map relies on, yeah, it's a mess, because we almost started in the same period. So we both started and we both grew a little bit towards the larger data sets, and now it's too late to change it. And so we discussed the whole thing and said, okay, it's too late, but as long as we're discussing almost the same thing with different names, who cares? It's data. If you want our data, if we want your data, it's always data. So anyway, the approach is slightly different. We are, with an open history map project, we want to have more control over the academic aspect of the data sets, and so we have the data index that you can reach at index.openhistorymap.org. And with that, we want to try to manage the complexity of the data and the data quality with a separate element. While open historical map tries to do that with the open street map approach, which is slightly different, more community-based, more structure-based. Our approach is more towards information that is verified and controlled by academia or academic procedures. We're not directly importing data. We're importing data for people who generate data. And on the other hand, the other great difference is that we want to track also ephemera information, meaning where was somebody at some point in time? What were the movements of one chip over time? And so with these information, this is our information that open historical map does not have and does not want to have because it's open street map of the past. And for that reason, we will most probably in the next month try to integrate that data from our database, from our interface, from our system into open historical map so that we can have a shared structure for event visualization and something like open... Let's say we also have something like historical street view where we look at historical photos and paintings and show them within their position in time and space. So that creates a whole set of information that, again, open historical map does not have and does not want to have while we try to use it for digital humanities and so becoming a more structured approach. Yeah, thank you, Mark, for this explanation about both projects. And we will end here this session. Thank you, Mark, for the presenters for being here and sharing their knowledge. We have the OSM General Meeting in a few minutes and you are invited to participate in the life of the association and the board is there and please come to the general meeting of OSGEO. Thank you all and see you around.
Open History Map, right? Yeah, exactly. Oh, good, started looking at... For this reason we had to develop an architecture that gives us the ability one of the items that are stored in the system, give us the map vector tile and the date data. By the way, the map is available here. If itself changes only in these instance, which partitioning as well for this infrastructure. And the caching of the whole infrastructure. And people on the humanistic side. It is not easy to data from a national level, national size data was somebody at some point in time? What were the