Establishing a Serverless Dynamic Tile Generation Workflow in Aws
Lynch, Callum
Establishing a Serverless Dynamic Tile Generation Workflow
 As I mentioned, my name is Carl Leach. I do work for the Australian Geospatial Talent and Social Development Organization for AGO. We're a part of the Department of Defence and we provide geospatial services to other parts of the department as well as to the Australian Defence Force. So today I'd like to talk to you about a project which my team and I took within our Amazon Web Services environment to devise a cloud-native approach to generating raster targets. So as a developer I do have a bit of a tendency to get the top-down data but today I'd like to keep the talk pretty high level. So most of the people in this room will have a pretty good understanding of what the project is going to talk about but I'll just give a quick primer for those who might be able to listen to me now. So I'll be talking a lot about AWS. So AWS of course is Amazon Web Services. So I'll talk about specific services. I'll try to define them clearly and in some cases I'll talk a little bit about why we might want to use them. I'll pick up here. How's that guys? Is that a bit better? Okay. So it'll also be helpful to have a basic knowledge of what I mean when I talk about raster tiles especially as they relate to TARD Web Services. So let me start by going over the problem. So last year the Department of Defense acquired a global imagery data set from Airbus. That data set provided high resolution imagery across the surface of the earth and it looks something like this. So obviously this is just a few of the highlights. In reality the world is a big place as we all know it takes a lot of data to cover it. In the case of the Airbus data set we're talking about 187 terabytes of data and that was in JPEG 2000 format. So our aim was to take this large data set and service enable it so that our customers could efficiently access imagery basically as a TARD Web Service. So most of our customers simply require this data as a backdrop to their operational planning and for situational awareness. So we're not talking about providing out the data itself but rather just a visualization. So a TARD Web Service will help us as well for this purpose. So how would we normally accomplish this? We'd probably normally look up using one of these guys in the case where our imagery data set is stored in our on-premises data arrays. So for example we might use Geo server to host the imagery and use something like Geo Web Cache sitting at the front end to allow us to manage the cache that results from that service. This is mostly stuff that most of us in this room have seen before. But we did run into a problem. So the Airbus data set was simply too large to store on our local data arrays. Instead we had it transferred into our Amazon cloud storage. So the specific service we're talking about here is Amazon's Simple Storage Service or S3. So S3 has a whole host of great features. It's one of AWS's core services. I won't spend too much time talking about the S3 features but what's important for us to know in the context of today's talk is that unlike in the situation where we have data stored on hard disk and we might be accessing it via, for example, the IEO protocol, with S3 all object interaction happens over HTTP and that will become important in a second. I want to take a step back for a moment and talk about JPEG 2000 data however. So JPEG 2000 data, unlike more cloud friendly formats like GeoTips or COGS, doesn't support range reading. And what this means in the context of generating a tile is to clip out an individual small tile from a source image. We must first load the source image in its entirety into local memory. So these source images for our Airbus data set range anywhere between about the 700 megabyte to 1.5 gigabyte size. So we're talking about fairly large files. So in a traditional scenario, this isn't an issue. We see IEO loading these source image files very quickly. It's a very fast protocol and as a result we're very unlikely to see any bottleneck forming at that stage. But when we put HTTP in the middle in a cloud scenario, we get a situation where we're going to start to see a very large bottleneck forming just in terms of reading the data across the network. So it pretty quickly became clear to us that these source image files of a gigabyte or more, we're just simply going to be too big to be able to efficiently transfer them around the network. So we needed to consider another strategy. And this is a basic visualisation of what we came up with. No prizes for guessing where that is. We created a simple GDL Python script to break the source images down into smaller imagery chunks. Those chunks were around the 20 megabyte mark which by trial and error we found to be best in terms of transferring them across the network. The script also created a post-gis database and basically put the metadata that was extracted from the source images into that database. And most importantly for our purposes here, it stored the information about the geospatial boundaries resulting imagery chunk that allowed fast querying later on in the process. So this is all well and good but generally speaking, pre-processing 187 terabytes of imagery in this way is pretty prohibitively slow. So we looked at doing this on our existing on premises enterprise infrastructure basically and we found that it would have taken about two months to complete this over 187 terabytes which was really too slow for our needs. But Cloud really came to the rescue for us here. So by using a technology in AWS called an auto scaling group, we were able to take a process which would have otherwise taken a couple of months and completed in just a couple of days. And really that comes down to the ability of cloud infrastructure to scale in response to demand. So the auto scaling group effectively describes a collection of servers or in Amazon speak EC2s. Each EC2 in this scenario was GPU optimized for the client. All of those EC2s were working in parallel in order to complete this task. So the way that we had it set up was that we created a queue, each item on the queue representing an individual object or a source image basically. Each server would pull down an item from the queue, generate the source imagery chunks for that source image and then move on to the next one when it was finished. So really the value add here was the scale of the infrastructure that we could leverage. So to give you an idea, throughout the runtime of this process, we actually used about 7.3 terabytes of virtual memory which is orders of magnitude higher than what we could have achieved in our traditional on-premises environment. Not to mention we didn't have to set anything up in our on-premises environment. We simply leveraged the service in AWS. So after this had completed, we had the source data in a structure that we could use it efficiently. The next step in the process was to consider how we were actually going to go about generating the tiles themselves. So to this end, we wrote a small Python script using Rasterio or Rasterio. The association seems to be a little bit controversial. The scripts took the geospatial bounds for a given tile and performed a geospatial lookup against the PostGIS index that we created from the database I mentioned earlier. That lookup then told the script which source imagery chunks needed to be pulled down in order to compose the given tile. Once that was done, it then pulled those source imagery chunks down and used Rasterio to merge and clip the data and the end result was a base64 encoded JPEG tile. So once we've got this working in a manner that we were happy with in our desktop environment, the next step was to consider how we were going to go about deploying it in a production scenario. Certainly one option we considered was to simply deploy it in the traditional manner onto a virtual server or a small cluster of virtual servers. But given that we were already working in the AWS environment and we had some interesting services at our disposal, we thought we could do a little bit better. So this brings us nicely to talking about Lambda. So AWS Lambda is a managed service that provides what's known as function as a service. So in other words, this means the developer brings some code, sets up some custom triggers and some custom outputs, and then AWS takes care of the underlying infrastructure required to run the function as well as the entire runtime environment for that function. In our case, using Lambda as a deployment mechanism offered a few distinct advantages. So firstly and most importantly, we were able to get a fairly significant performance advantage from using Lambda and that comes back to that idea of horizontal scaling. So with Lambda, we can easily run multiple thousands of tile generations in response to requests at any one given time. So it scales up automatically to match the load and scales itself down when it's not needed. So really this translates to a fairly significant performance increase for a WMTS service. Anybody who's worked with WMTS before knows that we're likely to see batches of tile requests coming in and we want to be able to generate all of those tiles in parallel if we can to make the performance of the services as high as possible and Lambda suits that need really well. The other thing to note with Lambda is that we only pay for the actual runtime of the function. So when Lambda receives a tile request in isolation, for example, it spins up some infrastructure in the background and then subsequently spins the infrastructure down when the tile generation is finished. So this makes it very cost efficient since it means that we only pay for the actual runtime, the actual time it takes to generate any given tile. Lastly, as with all serverless technologies, Lambda simply removes the need for us to manage our own infrastructure. So that means no health checks, no patching, no manual scaling. It's all taken care of on our behalf which makes our job as developers much, much easier. So the final step in the process once we had got the capability to a place where we could generate tiles on the fly was to expose it via HTTP. So we wanted to be able to take users HTTP requests from their software clients, for example, and generate tiles in response. Luckily, AWS has a service specifically for this requirement. So AWS's API gateway really does what it says on the box. So it receives HTTP requests, extracts the request parameters from that request and integrates them into the back end. So that's Lambda in our case. When Lambda then finishes and returns some information, API gateway handles that information, performs some transformation and then pumps out a HTTP response in response to the requester. So in addition to providing this classic HTTP front door to our capability, API gateway also offered the ability to transform incoming request parameters on the fly. And this was a feature that really came in handy for us when we were looking at supporting multiple tile schemas for our API. So we certainly could have taken care of the mathematics and the transformations required to transform between tile schemas within the Lambda itself. But what we found was that we had much better results if we black boxed the Lambda function and left it alone, let it do the tile generation in isolation and abstracted a lot of that transformation mathematics into the API itself. So API gateway made that really easy by providing a GUI in order to help us manage those transformations and really made it a lot easier, less laborious in terms of supporting all of those tile schemas. So putting it all together, I'm conscious of the time here. This is the final workflow. I realized there's a lot of information on this slide. But basically what I wanted to highlight was that the total time taken from when a user requests a tile from the API to when that tile is returned to them is less than two seconds. So we get pretty good performance out of this system. If you look closely here, you'll notice there's a few elements which I haven't covered in detail. Specifically I haven't talked about the tile caching strategy and I've really only paid attention to the indexation which is fairly complex behind the scenes. But my goal today was to give you an idea of some of the flagship services in AWS and how we can chain them together to create a system that's greater than some of its parts. So where to from here? The performance of the current solution is a little bit variable for our liking, particularly over areas of the world where we have a lot of overlapping source imagery. We'd like to have some strategies to improve the performance across the board but also to make it a little bit more consistent. So that could extend to multi-level tile caching, further tuning of our Lambda to sort of be more in line with what we need to use it for, as well as applying some smarts to our tile generation script in terms of making it generate tiles as quickly and efficiently for us as possible. One example of that is that we're looking at using the JPEG 2000 Kakadu drivers since we understand it may provide some performance benefits over the open JPEG implementation that we're currently using. We'd also like to increase the number of clients that this system supports and a big part of that will be fleshing out the API with some of those peripheral request parameters to enable us to be more in line with ODC standards. So that's all I have for today. I will return to that architecture slide but I'm happy to take any questions with the short amount of time we've got. Thank you. Thanks, Dylan. I was going to mention that Kakadu driver because I understand that it does support HTTP range requests. Right, well there you go. That's something we're definitely investigating at the moment. We were introduced to Kakadu fairly late in the process but we hear it's pretty good at this kind of stuff. I'm suspicious that the open JPEG one does too but it's not as fast as Kakadu. Right, right. Anyhow, we've got time for questions from the audience. So if anybody's got a question, just wait for me to come up with the microphone. I'll get my exercise going up here. Good day. The AWS Lambda deployment package has a pretty limited size on it and GDAL can quite quickly blow that right out of the water. Can you talk about how you managed to make the package small enough to get into a Lambda size deployment package? Yeah, I thought somebody might ask about this because you're absolutely right. So it is worth highlighting that there are a couple of cheats that you can do to get around that. So probably the biggest thing I'd highlight first of all is, and you might be aware of this, if you are deploying deployment packages into Lambda, you can increase the amount of the size of the deployment package that's accepted by uploading to S3 first and then providing that link. But yes, we did run into a lot of issues and I didn't really have time to go into detail here today. We ran into a lot of issues in terms of packaging Lambda. What we found worked best for us was using Docker as a staging environment for that. So we were able to use Docker and some pre-compiled GDAL builds to really cut down on what we actually wanted to include. But even then, we're running pretty close to the edge there. So our deployment package is running around the 60 megabyte mark, which is sort of pushing up on that. I have heard that there might be some consideration of AWS increasing that limit, but that watched this space. Yes. Hi. Do you generate all the tiles all at once, or do you use some sort of predictive algorithm to generate them, to pre-generate them on the fly? Yeah, good question. So basically, we generate a number of levels of tiles at the top. So we generate down to level 11 and cache all of that just for fast access. Then all of the tiles which are generated on the fly via HCE API request are then cached. So any tile that has been generated in the past can be quickly retrieved from the cache. All other tiles are generated on the fly, basically. Right at the top. Thanks for that. I just wanted to ask about the security of it. So is all this map like classified or unclassified, or how do you provide, like, is there different access for certain users, or providing tiles at certain levels? Yeah, good. I'm going to try to answer this. It's not my area, but I'll do my best. So first of all, all of this data is unclassified. It's sourced commercially. So while we can't distribute it because of commercial licensing, it is completely unclassified. So none of it's collected as part of our classified workflows. So that removes some of the issue there. But in terms of only providing access to certain users, we use IP range restrictions on the API. So previously we had been using IP range restrictions on S3. What isn't shown particularly well here is that all the tiles actually use S3 as the creation point before they're accessed. But we found that API gateway handled that way, way better. So just by restricting the IP ranges there, that's gone a long way towards doing this. What we do want to integrate in the future is ALDAF integration to allow only certain roles to access the data. But that's something that's going to probably come a little bit lighter for us.
 As I mentioned, my name is Carl Leach. I do work for the Australian Geospatial Talent  and Social Development Organization for AGO. We're a part of the Department of Defence and  we provide geospatial services to other parts of the department as well as to the Australian  Defence Force. So today I'd like to talk to you about a project which my team and I took  within our Amazon Web Services environment to devise a cloud-native approach to generating  raster targets. So as a developer I do have a bit of a tendency to get the top-down data  but today I'd like to keep the talk pretty high level. So most of the people in this room  will have a pretty good understanding of what the project is going to talk about but I'll  just give a quick primer for those who might be able to listen to me now. So I'll be talking  a lot about AWS. So AWS of course is Amazon Web Services. So I'll talk about specific  services. I'll try to define them clearly and in some cases I'll talk a little bit about  why we might want to use them. I'll pick up here.  How's that guys? Is that a bit better? Okay. So it'll also be helpful to have a basic  knowledge of what I mean when I talk about raster tiles especially as they relate to  TARD Web Services. So let me start by going over the problem. So last year the Department  of Defense acquired a global imagery data set from Airbus. That data set provided high  resolution imagery across the surface of the earth and it looks something like this. So  obviously this is just a few of the highlights. In reality the world is a big place as we  all know it takes a lot of data to cover it. In the case of the Airbus data set we're talking  about 187 terabytes of data and that was in JPEG 2000 format. So our aim was to take this  large data set and service enable it so that our customers could efficiently access imagery  basically as a TARD Web Service. So most of our customers simply require this data as a  backdrop to their operational planning and for situational awareness. So we're not talking  about providing out the data itself but rather just a visualization. So a TARD Web Service  will help us as well for this purpose. So how would we normally accomplish this? We'd  probably normally look up using one of these guys in the case where our imagery data set  is stored in our on-premises data arrays. So for example we might use Geo server to host  the imagery and use something like Geo Web Cache sitting at the front end to allow us  to manage the cache that results from that service. This is mostly stuff that most of  us in this room have seen before. But we did run into a problem. So the Airbus data set  was simply too large to store on our local data arrays. Instead we had it transferred  into our Amazon cloud storage. So the specific service we're talking about here is Amazon's  Simple Storage Service or S3. So S3 has a whole host of great features. It's one of  AWS's core services. I won't spend too much time talking about the S3 features but what's  important for us to know in the context of today's talk is that unlike in the situation  where we have data stored on hard disk and we might be accessing it via, for example,  the IEO protocol, with S3 all object interaction happens over HTTP and that will become important  in a second. I want to take a step back for a moment and talk about JPEG 2000 data however.  So JPEG 2000 data, unlike more cloud friendly formats like GeoTips or COGS, doesn't support  range reading. And what this means in the context of generating a tile is to clip out  an individual small tile from a source image. We must first load the source image in its  entirety into local memory. So these source images for our Airbus data set range anywhere  between about the 700 megabyte to 1.5 gigabyte size. So we're talking about fairly large  files. So in a traditional scenario, this isn't an issue. We see IEO loading these source  image files very quickly. It's a very fast protocol and as a result we're very unlikely  to see any bottleneck forming at that stage. But when we put HTTP in the middle in a cloud  scenario, we get a situation where we're going to start to see a very large bottleneck forming  just in terms of reading the data across the network. So it pretty quickly became clear  to us that these source image files of a gigabyte or more, we're just simply going to be too  big to be able to efficiently transfer them around the network. So we needed to consider  another strategy. And this is a basic visualisation of what we came up with. No prizes for guessing  where that is. We created a simple GDL Python script to break the source images down into  smaller imagery chunks. Those chunks were around the 20 megabyte mark which by trial  and error we found to be best in terms of transferring them across the network. The script  also created a post-gis database and basically put the metadata that was extracted from the  source images into that database. And most importantly for our purposes here, it stored  the information about the geospatial boundaries resulting imagery chunk that allowed fast  querying later on in the process. So this is all well and good but generally speaking,  pre-processing 187 terabytes of imagery in this way is pretty prohibitively slow. So  we looked at doing this on our existing on premises enterprise infrastructure basically  and we found that it would have taken about two months to complete this over 187 terabytes  which was really too slow for our needs. But Cloud really came to the rescue for us here.  So by using a technology in AWS called an auto scaling group, we were able to take a  process which would have otherwise taken a couple of months and completed in just a  couple of days. And really that comes down to the ability of cloud infrastructure to  scale in response to demand. So the auto scaling group effectively describes a collection of  servers or in Amazon speak EC2s. Each EC2 in this scenario was GPU optimized for the  client. All of those EC2s were working in parallel in order to complete this task. So  the way that we had it set up was that we created a queue, each item on the queue representing  an individual object or a source image basically. Each server would pull down an item from the  queue, generate the source imagery chunks for that source image and then move on to the  next one when it was finished. So really the value add here was the scale of the infrastructure  that we could leverage. So to give you an idea, throughout the runtime of this process,  we actually used about 7.3 terabytes of virtual memory which is orders of magnitude higher  than what we could have achieved in our traditional on-premises environment. Not to mention we  didn't have to set anything up in our on-premises environment. We simply leveraged the service  in AWS. So after this had completed, we had the source data in a structure that we could  use it efficiently. The next step in the process was to consider how we were actually going  to go about generating the tiles themselves. So to this end, we wrote a small Python script  using Rasterio or Rasterio. The association seems to be a little bit controversial. The  scripts took the geospatial bounds for a given tile and performed a geospatial lookup against  the PostGIS index that we created from the database I mentioned earlier. That lookup  then told the script which source imagery chunks needed to be pulled down in order to  compose the given tile. Once that was done, it then pulled those source imagery chunks  down and used Rasterio to merge and clip the data and the end result was a base64 encoded  JPEG tile. So once we've got this working in a manner that we were happy with in our  desktop environment, the next step was to consider how we were going to go about deploying it  in a production scenario. Certainly one option we considered was to simply deploy it in the  traditional manner onto a virtual server or a small cluster of virtual servers. But given  that we were already working in the AWS environment and we had some interesting services at our  disposal, we thought we could do a little bit better. So this brings us nicely to talking  about Lambda. So AWS Lambda is a managed service that provides what's known as function as  a service. So in other words, this means the developer brings some code, sets up some custom  triggers and some custom outputs, and then AWS takes care of the underlying infrastructure  required to run the function as well as the entire runtime environment for that function.  In our case, using Lambda as a deployment mechanism offered a few distinct advantages.  So firstly and most importantly, we were able to get a fairly significant performance advantage  from using Lambda and that comes back to that idea of horizontal scaling. So with Lambda,  we can easily run multiple thousands of tile generations in response to requests at any  one given time. So it scales up automatically to match the load and scales itself down when  it's not needed. So really this translates to a fairly significant performance increase  for a WMTS service. Anybody who's worked with WMTS before knows that we're likely to see  batches of tile requests coming in and we want to be able to generate all of those tiles in  parallel if we can to make the performance of the services as high as possible and Lambda  suits that need really well. The other thing to note with Lambda is that we only pay for  the actual runtime of the function. So when Lambda receives a tile request in isolation,  for example, it spins up some infrastructure in the background and then subsequently spins  the infrastructure down when the tile generation is finished. So this makes it very cost efficient  since it means that we only pay for the actual runtime, the actual time it takes to generate  any given tile. Lastly, as with all serverless technologies, Lambda simply removes the need  for us to manage our own infrastructure. So that means no health checks, no patching, no manual  scaling. It's all taken care of on our behalf which makes our job as developers much, much easier.  So the final step in the process once we had got the capability to a place where we could  generate tiles on the fly was to expose it via HTTP. So we wanted to be able to take users  HTTP requests from their software clients, for example, and generate tiles in response.  Luckily, AWS has a service specifically for this requirement. So AWS's API gateway really  does what it says on the box. So it receives HTTP requests, extracts the request parameters  from that request and integrates them into the back end. So that's Lambda in our case.  When Lambda then finishes and returns some information, API gateway handles that information,  performs some transformation and then pumps out a HTTP response in response to the requester.  So in addition to providing this classic HTTP front door to our capability, API gateway  also offered the ability to transform incoming request parameters on the fly.  And this was a feature that really came in handy for us when we were looking at supporting  multiple tile schemas for our API. So we certainly could have taken care of the mathematics and  the transformations required to transform between tile schemas within the Lambda itself.  But what we found was that we had much better results if we black boxed the Lambda function  and left it alone, let it do the tile generation in isolation and abstracted a lot of that  transformation mathematics into the API itself. So API gateway made that really easy by providing  a GUI in order to help us manage those transformations and really made it a lot easier, less laborious  in terms of supporting all of those tile schemas. So putting it all together, I'm conscious  of the time here. This is the final workflow. I realized there's a lot of information on  this slide. But basically what I wanted to highlight was that the total time taken from  when a user requests a tile from the API to when that tile is returned to them is less  than two seconds. So we get pretty good performance out of this system.  If you look closely here, you'll notice there's a few elements which I haven't covered in detail.  Specifically I haven't talked about the tile caching strategy and I've really only paid  attention to the indexation which is fairly complex behind the scenes. But my goal today  was to give you an idea of some of the flagship services in AWS and how we can chain them  together to create a system that's greater than some of its parts. So where to from here?  The performance of the current solution is a little bit variable for our liking, particularly  over areas of the world where we have a lot of overlapping source imagery. We'd like to  have some strategies to improve the performance across the board but also to make it a little  bit more consistent. So that could extend to multi-level tile caching, further tuning  of our Lambda to sort of be more in line with what we need to use it for, as well as applying  some smarts to our tile generation script in terms of making it generate tiles as quickly  and efficiently for us as possible. One example of that is that we're looking at using the  JPEG 2000 Kakadu drivers since we understand it may provide some performance benefits over  the open JPEG implementation that we're currently using. We'd also like to increase the number  of clients that this system supports and a big part of that will be fleshing out the API  with some of those peripheral request parameters to enable us to be more in line with ODC standards.  So that's all I have for today. I will return to that architecture slide but I'm happy to  take any questions with the short amount of time we've got. Thank you.  Thanks, Dylan.  I was going to mention that Kakadu driver because I understand that it does support  HTTP range requests. Right, well there you go. That's something we're definitely investigating  at the moment. We were introduced to Kakadu fairly late in the process but we hear it's  pretty good at this kind of stuff. I'm suspicious that the open JPEG one does too but it's not  as fast as Kakadu. Right, right. Anyhow, we've got time for questions from the audience.  So if anybody's got a question, just wait for me to come up with the microphone. I'll get  my exercise going up here.  Good day. The AWS Lambda deployment package has a pretty limited size on it and GDAL can  quite quickly blow that right out of the water. Can you talk about how you managed to make  the package small enough to get into a Lambda size deployment package?  Yeah, I thought somebody might ask about this because you're absolutely right. So it is worth  highlighting that there are a couple of cheats that you can do to get around that. So probably  the biggest thing I'd highlight first of all is, and you might be aware of this, if you are  deploying deployment packages into Lambda, you can increase the amount of the size of  the deployment package that's accepted by uploading to S3 first and then providing that link.  But yes, we did run into a lot of issues and I didn't really have time to go into detail here today.  We ran into a lot of issues in terms of packaging Lambda. What we found worked best for us was  using Docker as a staging environment for that. So we were able to use Docker and some pre-compiled  GDAL builds to really cut down on what we actually wanted to include. But even then, we're running  pretty close to the edge there. So our deployment package is running around the 60 megabyte mark,  which is sort of pushing up on that. I have heard that there might be some consideration of AWS  increasing that limit, but that watched this space.  Yes. Hi. Do you generate all the tiles all at once, or do you use some sort of predictive  algorithm to generate them, to pre-generate them on the fly?  Yeah, good question. So basically, we generate a number of levels of tiles at the top. So we  generate down to level 11 and cache all of that just for fast access. Then all of the tiles which  are generated on the fly via HCE API request are then cached. So any tile that has been generated  in the past can be quickly retrieved from the cache. All other tiles are generated on the fly,  basically.  Right at the top. Thanks for that.  I just wanted to ask about the security of it. So is all this map like classified or  unclassified, or how do you provide, like, is there different access for certain users,  or providing tiles at certain levels?  Yeah, good. I'm going to try to answer this. It's not my area, but I'll do my best. So first of all,  all of this data is unclassified. It's sourced commercially. So while we can't distribute it  because of commercial licensing, it is completely unclassified. So none of it's collected as part  of our classified workflows. So that removes some of the issue there. But in terms of only  providing access to certain users, we use IP range restrictions on the API. So previously we had  been using IP range restrictions on S3. What isn't shown particularly well here is that all the  tiles actually use S3 as the creation point before they're accessed. But we found that API  gateway handled that way, way better. So just by restricting the IP ranges there, that's gone a  long way towards doing this. What we do want to integrate in the future is ALDAF integration to  allow only certain roles to access the data. But that's something that's going to probably come  a little bit lighter for us.
Callum_Lynch__Establishing_a_Serverless_Dynamic_Tile_Generation_Workflow_HlunDRbIrow_0_59.png Callum_Lynch__Establishing_a_Serverless_Dynamic_Tile_Generation_Workflow_HlunDRbIrow_60_743.png Callum_Lynch__Establishing_a_Serverless_Dynamic_Tile_Generation_Workflow_HlunDRbIrow_744_2111.png Callum_Lynch__Establishing_a_Serverless_Dynamic_Tile_Generation_Workflow_HlunDRbIrow_2112_2507.png Callum_Lynch__Establishing_a_Serverless_Dynamic_Tile_Generation_Workflow_HlunDRbIrow_2508_2735.png Callum_Lynch__Establishing_a_Serverless_Dynamic_Tile_Generation_Workflow_HlunDRbIrow_2736_2843.png Callum_Lynch__Establishing_a_Serverless_Dynamic_Tile_Generation_Workflow_HlunDRbIrow_2844_3515.png Callum_Lynch__Establishing_a_Serverless_Dynamic_Tile_Generation_Workflow_HlunDRbIrow_3516_4367.png Callum_Lynch__Establishing_a_Serverless_Dynamic_Tile_Generation_Workflow_HlunDRbIrow_4368_6119.png Callum_Lynch__Establishing_a_Serverless_Dynamic_Tile_Generation_Workflow_HlunDRbIrow_6120_6419.png Callum_Lynch__Establishing_a_Serverless_Dynamic_Tile_Generation_Workflow_HlunDRbIrow_6420_6791.png Callum_Lynch__Establishing_a_Serverless_Dynamic_Tile_Generation_Workflow_HlunDRbIrow_6792_7043.png Callum_Lynch__Establishing_a_Serverless_Dynamic_Tile_Generation_Workflow_HlunDRbIrow_7044_7139.png Callum_Lynch__Establishing_a_Serverless_Dynamic_Tile_Generation_Workflow_HlunDRbIrow_7140_8855.png Callum_Lynch__Establishing_a_Serverless_Dynamic_Tile_Generation_Workflow_HlunDRbIrow_8856_10907.png Callum_Lynch__Establishing_a_Serverless_Dynamic_Tile_Generation_Workflow_HlunDRbIrow_10908_11231.png Callum_Lynch__Establishing_a_Serverless_Dynamic_Tile_Generation_Workflow_HlunDRbIrow_11232_12983.png Callum_Lynch__Establishing_a_Serverless_Dynamic_Tile_Generation_Workflow_HlunDRbIrow_12984_16007.png Callum_Lynch__Establishing_a_Serverless_Dynamic_Tile_Generation_Workflow_HlunDRbIrow_16008_16607.png Callum_Lynch__Establishing_a_Serverless_Dynamic_Tile_Generation_Workflow_HlunDRbIrow_16608_18743.png Callum_Lynch__Establishing_a_Serverless_Dynamic_Tile_Generation_Workflow_HlunDRbIrow_18744_19811.png Callum_Lynch__Establishing_a_Serverless_Dynamic_Tile_Generation_Workflow_HlunDRbIrow_19812_21335.png Callum_Lynch__Establishing_a_Serverless_Dynamic_Tile_Generation_Workflow_HlunDRbIrow_21336_21455.png Callum_Lynch__Establishing_a_Serverless_Dynamic_Tile_Generation_Workflow_HlunDRbIrow_21456_21503.png Callum_Lynch__Establishing_a_Serverless_Dynamic_Tile_Generation_Workflow_HlunDRbIrow_21504_21551.png Callum_Lynch__Establishing_a_Serverless_Dynamic_Tile_Generation_Workflow_HlunDRbIrow_21552_21611.png Callum_Lynch__Establishing_a_Serverless_Dynamic_Tile_Generation_Workflow_HlunDRbIrow_21612_28007.png Callum_Lynch__Establishing_a_Serverless_Dynamic_Tile_Generation_Workflow_HlunDRbIrow_28008_28058.png
As I mentioned, my name is Carl Leach. I do work for the Australian Geospatial Talent and Social Development Organization for AGO. We're a part of the Department of Defence and we provide geospatial services to other parts of the department as well as to the Australian Defence Force. So today I'd like to talk to you about a project which my team and I took within our Amazon Web Services environment to devise a cloud-native approach to generating raster targets. So as a developer I do have a bit of a tendency to get the top-down data but today I'd like to keep the talk pretty high level. So most of the people in this room will have a pretty good understanding of what the project is going to talk about but I'll just give a quick primer for those who might be able to listen to me now. So I'll be talking a lot about AWS. So AWS of course is Amazon Web Services. So I'll talk about specific services. I'll try to define them clearly and in some cases I'll talk a little bit about why we might want to use them. I'll pick up here. How's that guys? Is that a bit better? Okay. So it'll also be helpful to have a basic knowledge of what I mean when I talk about raster tiles especially as they relate to TARD Web Services. So let me start by going over the problem. So last year the Department of Defense acquired a global imagery data set from Airbus. That data set provided high resolution imagery across the surface of the earth and it looks something like this. So obviously this is just a few of the highlights. In reality the world is a big place as we all know it takes a lot of data to cover it. In the case of the Airbus data set we're talking about 187 terabytes of data and that was in JPEG 2000 format. So our aim was to take this large data set and service enable it so that our customers could efficiently access imagery basically as a TARD Web Service. So most of our customers simply require this data as a backdrop to their operational planning and for situational awareness. So we're not talking about providing out the data itself but rather just a visualization. So a TARD Web Service will help us as well for this purpose. So how would we normally accomplish this? We'd probably normally look up using one of these guys in the case where our imagery data set is stored in our on-premises data arrays. So for example we might use Geo server to host the imagery and use something like Geo Web Cache sitting at the front end to allow us to manage the cache that results from that service. This is mostly stuff that most of us in this room have seen before. But we did run into a problem. So the Airbus data set was simply too large to store on our local data arrays. Instead we had it transferred into our Amazon cloud storage. So the specific service we're talking about here is Amazon's Simple Storage Service or S3. So S3 has a whole host of great features. It's one of AWS's core services. I won't spend too much time talking about the S3 features but what's important for us to know in the context of today's talk is that unlike in the situation where we have data stored on hard disk and we might be accessing it via, for example, the IEO protocol, with S3 all object interaction happens over HTTP and that will become important in a second. I want to take a step back for a moment and talk about JPEG 2000 data however. So JPEG 2000 data, unlike more cloud friendly formats like GeoTips or COGS, doesn't support range reading. And what this means in the context of generating a tile is to clip out an individual small tile from a source image. We must first load the source image in its entirety into local memory. So these source images for our Airbus data set range anywhere between about the 700 megabyte to 1.5 gigabyte size. So we're talking about fairly large files. So in a traditional scenario, this isn't an issue. We see IEO loading these source image files very quickly. It's a very fast protocol and as a result we're very unlikely to see any bottleneck forming at that stage. But when we put HTTP in the middle in a cloud scenario, we get a situation where we're going to start to see a very large bottleneck forming just in terms of reading the data across the network. So it pretty quickly became clear to us that these source image files of a gigabyte or more, we're just simply going to be too big to be able to efficiently transfer them around the network. So we needed to consider another strategy. And this is a basic visualisation of what we came up with. No prizes for guessing where that is. We created a simple GDL Python script to break the source images down into smaller imagery chunks. Those chunks were around the 20 megabyte mark which by trial and error we found to be best in terms of transferring them across the network. The script also created a post-gis database and basically put the metadata that was extracted from the source images into that database. And most importantly for our purposes here, it stored the information about the geospatial boundaries resulting imagery chunk that allowed fast querying later on in the process. So this is all well and good but generally speaking, pre-processing 187 terabytes of imagery in this way is pretty prohibitively slow. So we looked at doing this on our existing on premises enterprise infrastructure basically and we found that it would have taken about two months to complete this over 187 terabytes which was really too slow for our needs. But Cloud really came to the rescue for us here. So by using a technology in AWS called an auto scaling group, we were able to take a process which would have otherwise taken a couple of months and completed in just a couple of days. And really that comes down to the ability of cloud infrastructure to scale in response to demand. So the auto scaling group effectively describes a collection of servers or in Amazon speak EC2s. Each EC2 in this scenario was GPU optimized for the client. All of those EC2s were working in parallel in order to complete this task. So the way that we had it set up was that we created a queue, each item on the queue representing an individual object or a source image basically. Each server would pull down an item from the queue, generate the source imagery chunks for that source image and then move on to the next one when it was finished. So really the value add here was the scale of the infrastructure that we could leverage. So to give you an idea, throughout the runtime of this process, we actually used about 7.3 terabytes of virtual memory which is orders of magnitude higher than what we could have achieved in our traditional on-premises environment. Not to mention we didn't have to set anything up in our on-premises environment. We simply leveraged the service in AWS. So after this had completed, we had the source data in a structure that we could use it efficiently. The next step in the process was to consider how we were actually going to go about generating the tiles themselves. So to this end, we wrote a small Python script using Rasterio or Rasterio. The association seems to be a little bit controversial. The scripts took the geospatial bounds for a given tile and performed a geospatial lookup against the PostGIS index that we created from the database I mentioned earlier. That lookup then told the script which source imagery chunks needed to be pulled down in order to compose the given tile. Once that was done, it then pulled those source imagery chunks down and used Rasterio to merge and clip the data and the end result was a base64 encoded JPEG tile. So once we've got this working in a manner that we were happy with in our desktop environment, the next step was to consider how we were going to go about deploying it in a production scenario. Certainly one option we considered was to simply deploy it in the traditional manner onto a virtual server or a small cluster of virtual servers. But given that we were already working in the AWS environment and we had some interesting services at our disposal, we thought we could do a little bit better. So this brings us nicely to talking about Lambda. So AWS Lambda is a managed service that provides what's known as function as a service. So in other words, this means the developer brings some code, sets up some custom triggers and some custom outputs, and then AWS takes care of the underlying infrastructure required to run the function as well as the entire runtime environment for that function. In our case, using Lambda as a deployment mechanism offered a few distinct advantages. So firstly and most importantly, we were able to get a fairly significant performance advantage from using Lambda and that comes back to that idea of horizontal scaling. So with Lambda, we can easily run multiple thousands of tile generations in response to requests at any one given time. So it scales up automatically to match the load and scales itself down when it's not needed. So really this translates to a fairly significant performance increase for a WMTS service. Anybody who's worked with WMTS before knows that we're likely to see batches of tile requests coming in and we want to be able to generate all of those tiles in parallel if we can to make the performance of the services as high as possible and Lambda suits that need really well. The other thing to note with Lambda is that we only pay for the actual runtime of the function. So when Lambda receives a tile request in isolation, for example, it spins up some infrastructure in the background and then subsequently spins the infrastructure down when the tile generation is finished. So this makes it very cost efficient since it means that we only pay for the actual runtime, the actual time it takes to generate any given tile. Lastly, as with all serverless technologies, Lambda simply removes the need for us to manage our own infrastructure. So that means no health checks, no patching, no manual scaling. It's all taken care of on our behalf which makes our job as developers much, much easier. So the final step in the process once we had got the capability to a place where we could generate tiles on the fly was to expose it via HTTP. So we wanted to be able to take users HTTP requests from their software clients, for example, and generate tiles in response. Luckily, AWS has a service specifically for this requirement. So AWS's API gateway really does what it says on the box. So it receives HTTP requests, extracts the request parameters from that request and integrates them into the back end. So that's Lambda in our case. When Lambda then finishes and returns some information, API gateway handles that information, performs some transformation and then pumps out a HTTP response in response to the requester. So in addition to providing this classic HTTP front door to our capability, API gateway also offered the ability to transform incoming request parameters on the fly. And this was a feature that really came in handy for us when we were looking at supporting multiple tile schemas for our API. So we certainly could have taken care of the mathematics and the transformations required to transform between tile schemas within the Lambda itself. But what we found was that we had much better results if we black boxed the Lambda function and left it alone, let it do the tile generation in isolation and abstracted a lot of that transformation mathematics into the API itself. So API gateway made that really easy by providing a GUI in order to help us manage those transformations and really made it a lot easier, less laborious in terms of supporting all of those tile schemas. So putting it all together, I'm conscious of the time here. This is the final workflow. I realized there's a lot of information on this slide. But basically what I wanted to highlight was that the total time taken from when a user requests a tile from the API to when that tile is returned to them is less than two seconds. So we get pretty good performance out of this system. If you look closely here, you'll notice there's a few elements which I haven't covered in detail. Specifically I haven't talked about the tile caching strategy and I've really only paid attention to the indexation which is fairly complex behind the scenes. But my goal today was to give you an idea of some of the flagship services in AWS and how we can chain them together to create a system that's greater than some of its parts. So where to from here? The performance of the current solution is a little bit variable for our liking, particularly over areas of the world where we have a lot of overlapping source imagery. We'd like to have some strategies to improve the performance across the board but also to make it a little bit more consistent. So that could extend to multi-level tile caching, further tuning of our Lambda to sort of be more in line with what we need to use it for, as well as applying some smarts to our tile generation script in terms of making it generate tiles as quickly and efficiently for us as possible. One example of that is that we're looking at using the JPEG 2000 Kakadu drivers since we understand it may provide some performance benefits over the open JPEG implementation that we're currently using. We'd also like to increase the number of clients that this system supports and a big part of that will be fleshing out the API with some of those peripheral request parameters to enable us to be more in line with ODC standards. So that's all I have for today. I will return to that architecture slide but I'm happy to take any questions with the short amount of time we've got. Thank you. Thanks, Dylan. I was going to mention that Kakadu driver because I understand that it does support HTTP range requests. Right, well there you go. That's something we're definitely investigating at the moment. We were introduced to Kakadu fairly late in the process but we hear it's pretty good at this kind of stuff. I'm suspicious that the open JPEG one does too but it's not as fast as Kakadu. Right, right. Anyhow, we've got time for questions from the audience. So if anybody's got a question, just wait for me to come up with the microphone. I'll get my exercise going up here. Good day. The AWS Lambda deployment package has a pretty limited size on it and GDAL can quite quickly blow that right out of the water. Can you talk about how you managed to make the package small enough to get into a Lambda size deployment package? Yeah, I thought somebody might ask about this because you're absolutely right. So it is worth highlighting that there are a couple of cheats that you can do to get around that. So probably the biggest thing I'd highlight first of all is, and you might be aware of this, if you are deploying deployment packages into Lambda, you can increase the amount of the size of the deployment package that's accepted by uploading to S3 first and then providing that link. But yes, we did run into a lot of issues and I didn't really have time to go into detail here today. We ran into a lot of issues in terms of packaging Lambda. What we found worked best for us was using Docker as a staging environment for that. So we were able to use Docker and some pre-compiled GDAL builds to really cut down on what we actually wanted to include. But even then, we're running pretty close to the edge there. So our deployment package is running around the 60 megabyte mark, which is sort of pushing up on that. I have heard that there might be some consideration of AWS increasing that limit, but that watched this space. Yes. Hi. Do you generate all the tiles all at once, or do you use some sort of predictive algorithm to generate them, to pre-generate them on the fly? Yeah, good question. So basically, we generate a number of levels of tiles at the top. So we generate down to level 11 and cache all of that just for fast access. Then all of the tiles which are generated on the fly via HCE API request are then cached. So any tile that has been generated in the past can be quickly retrieved from the cache. All other tiles are generated on the fly, basically. Right at the top. Thanks for that. I just wanted to ask about the security of it. So is all this map like classified or unclassified, or how do you provide, like, is there different access for certain users, or providing tiles at certain levels? Yeah, good. I'm going to try to answer this. It's not my area, but I'll do my best. So first of all, all of this data is unclassified. It's sourced commercially. So while we can't distribute it because of commercial licensing, it is completely unclassified. So none of it's collected as part of our classified workflows. So that removes some of the issue there. But in terms of only providing access to certain users, we use IP range restrictions on the API. So previously we had been using IP range restrictions on S3. What isn't shown particularly well here is that all the tiles actually use S3 as the creation point before they're accessed. But we found that API gateway handled that way, way better. So just by restricting the IP ranges there, that's gone a long way towards doing this. What we do want to integrate in the future is ALDAF integration to allow only certain roles to access the data. But that's something that's going to probably come a little bit lighter for us.
Amazon Web Services environment to devise a