Running Geospatial Workloads on AWS
Roach, Craig
Craig's talk was the first talk in the "Cloud / Development" session at FOSS4G SotM Oceania 2019, organised by OSGeo Oceania and held at The National Library in Wellington, New Zealand from November 12-15 2019. FOSS4G SotM Oceania is the coming together of Oceania's geospatial open source and open data community - with four days of workshops, presentations, a community sprint and social events.
 Hi everyone. So what I wanted to talk to you about first of all is just explain what is the difference between running a geospatial workload on the cloud as opposed to on-premises or any other type of infrastructure because you're probably aware that if you were running a workload on any cloud, you could stand up a bunch of virtual machines and provision a bunch of block storage and everything would look exactly the same as any other environment. But there's one picture I'd like you to take away with you when you leave here and that's that the cloud gives you the option of flipping that whole infrastructure on its head. So instead of actually bringing the data to the compute infrastructure, you basically bring the compute infrastructure to the data. And in the AWS environment, that data is represented using S3 object stores. And the concept there is that S3 is a highly durable, highly scalable object storage layer that pretty much any AWS service will be able to integrate with natively. And so I've got Amazon Athena there. You can run SQL queries directly against CSV files or JSON files or Parquet files sitting on S3. You don't need to load them into a database. You can run Apache Spark jobs on them. You can do map reduce type tasks on S3 directly without moving it from S3 into a Hadoop file system. You can have Amazon EC2 instances directly referencing the S3 data. And I'll show you an example a little bit later where we've got a bunch of data, the NREL organization running HD5 data, sorry, HDF5 sitting on S3 and directly accessing it through EC2 instances, again without moving it to block storage. So that's probably the key differences because you're basically bringing the compute to the storage. It takes a lot of the plumbing and a lot of the hassle out of dealing with data. So the key phrase there is share, don't copy. By sharing the data, by publishing the data once from the producers and actually consuming it in number of times globally across your choice of infrastructure, whether it's work spaces for running say a heavy duty visualization software stack that needs multiple GPUs and 4K monitors and hundreds of gigabytes of RAM you can run that on a virtual desktop. You can do EC2 instances, map reducers I mentioned. You can do serverless compute with Lambda where you actually don't actually stand up servers at all. You write a piece of Python code, you put in pandas or your choice of geospatial libraries and directly reference the S3 data. Or you can do machine learning training based on the S3 data as well. And there's a bunch of customers of AWS in New Zealand, in Wellington in fact, doing a whole lot of machine learning work with large image files and large personal identifiable information sets as well. So the reason we, what I want to talk about specifically though is about open data. So taking that a step further, once we're actually talking about sharing data and bringing the compute, it makes it very attractive for organizations to be able to publish their data and share it using that S3 layer. And what we've been hearing from customers like Met Service and Digital Globes, it'd be really great if someone could actually take the problem of having to manage big FTP servers or file servers or web servers for distributing gigabytes or even terabytes of data a day away from us and us having to ensure availability. And likewise, the consumers of that data are telling us that what they really want is to avoid that hassle of having to do regular downloads. When I was at Met Service, that was a big part of our job was actually downloading global meteorological files from ECMWF, from the UK Met Office, from GFS, from Japanese Meteorological Agency, satellite data all over the infrastructure, all over the network infrastructure. That was almost like our number one headache. Compared to that, running the weather models was relatively simple, being a bit flippant, but it's a big job. So what AWS has responded to is by saying, well, why don't you let us actually manage that storage for you? And if the data is actually genuinely useful to a large number of people, we'll take on the actual storage costs and the data access costs as well. So the only thing you need to do is basically provide, as a producer, is provide the content. So probably the biggest advantage is that it opens up your data as a producer to a larger community of users. We have a saying in AWS that 99% of the smartest people in the world do not work for AWS. They work for someone else. And that's probably true of your organizations as well, maybe not as much. But it's certainly the case that most of the brains actually don't work for you. And you can leverage that by publishing your data. And I'll go a little bit further than that. It's actually a lot more than just dumping the data somewhere on S3. But by publishing that data, you open it up to that community. It lowers the cost of research as well, because if you're not actually having to worry about network pipes and that sort of thing, if you're able to run Jupyter Notebooks or write your own Python code directly against the S3 data, it means you're actually running projects in a matter of minutes and you're not having to invest in infrastructure as well. So I'll talk to you a little bit about our public data sets already in our registry. These are some of the major providers of those data sets. As you imagine, NASA publishes a lot of satellite imagery. You can see the Hubble Space Telescope data there if you're interested in it as well. But there's some surprising organisations there which probably aren't geospatial in nature, but are national institutes of health. They publish de-identified patient records into this repository as well. There's financial data. There's a stack of meteorological data sources ranging from radar, satellite imagery, global weather models, local area models like Waf and that sort of thing. Now in terms of actually how you go about finding it, if you go to the AWS, there's a URL there, open data at AWS, you'll see there's a registry and you can basically just put in a search and that will give you all the information, a link to the data set and it'll have a full page about what's actually, just back up, a full page about how to access that data and it's usually just a URL. But more importantly, there's robust documentation that sits behind it. There's tutorials, there's Jupyter notebooks, there's GitHub links where you can actually access some tools that will make it easy to consume the data as well. So in terms of how to share data, one of the things we've learnt through this open data program is that there's more than just dropping the data somewhere and saying go for your lives and make use of this. There's a few things you need to do. The first thing is non-technical considerations associated with sharing. Number one is actually trust. If someone's going to invest time in building a pipeline or a data processing pipeline against your data, they need to be able to trust your data and that means that if you've got any sort of remote sensing network with actual mechanical components or cameras or whatever, those things go wrong. So when they go wrong, you need to be able to tell people that it's gone wrong so they can actually adapt to that. Landsat's a really good example of that, that they actually had some sensors fail on a satellite a few years ago and they actually went bent over backwards to make sure that everyone knew about that and everyone was factoring in those sensor failures into their software stacks. We used to have that with MetService as well with some of the polar orbiters that over time they would break down, sensors would get irradiated and they would stop working or lose sensitivity. And as a provider, they were obliged to, they told us that and we were able to incorporate that. So that really builds trust. Second thing is actually exactly the same as software, you need good documentation, you need use cases, you need tutorials, you need sample code that people can actually get started with. And lastly, accessibility. Again, if people are going to put a lot of effort into tooling, they want to know they can get that data when they need it, particularly if they're doing operational processing. Some of these meteorological agencies are publishing hourly observations on this repository. It's not just static data, it's actually live data. Some data sets are being updated every minute. So if you've got consumers or if you are a consumer and you're consuming that data, you want to know that that data is always there and you can get to it. And it's not going to grind to a halt when lots of people are downloading it. So just that little curve there, it means that what that's suggesting is that as the trustworthiness level of documentation and accessibility of your data resource improves, your user base basically starts to grow pretty steeply. If the data is really useful, people will persist and they'll get onto it and they'll figure it out somehow. But if you're really going to access a large user base and particularly busy people, you need to invest a bit of time. The second thing is technical considerations. And this is a little bit caranturative, but you can overthink and as well as underthink your data representation formats. Underthink is pretty obvious. It's when you just take a binary database dump or something like that, drop it onto S3 and say go for your lives. That's no good to anyone. Some people will access it, but it's going to be very difficult for them. You can also overthink it. If you design a handcraft or really fine-tuned API for a specific use case, you might think you're doing people a favour and for some of the user base you are, but you're also going to alienate some people that actually sit in the middle that need something a little bit more flexible than originally defined API, but don't want to go and read binary database dumps. So ideally what you're doing is actually trying to cater for multiple use cases. By all means, build your API, but also make the data available in a more raw format so that people can actually download it and do their own manipulations and slice it and dice it in the ways they make sense for them. So some common use cases, you can also optimise your data for some common use cases. I'll show you an example with Wind Farm modelling in a couple of minutes where the provider has actually gone through a techno economic analysis of the weather data. So they converted wind speeds into actual shaft power outputs for the turbines. So local agencies or power authorities could make decisions about how useful it would be to put a side of wind turbine in that location. So moving from the meteorological domain to the economic domain. Build a map in an index. Don't just drop it into a, don't just have a directory and trust that people will understand when you file naming conventions. Build a searchable index. Provide a cherry picking mechanism. So if you've built an index and people are able to target a slice of data or build a query that can select a subset, build a mechanism by which they can extract that data using range type selections. Because they'll save them a lot of time, they'll save them a lot of download time as well. And lastly, track usage. Measure how people are using your data. So I'll just finish up with a couple of examples. So one of them is on a common data repository. One of them is called the Blue Dot Observatory. And what that is is it's a consortium that's set up to analyse the Sentinel-2 satellite observation data. And their objective is actually to look at patterns of water consumption and actually the depletion of water resources throughout the world, particularly in the developing world. And there's a pretty scary map showing that could be Sydney. If anyone's from Sydney, that could be Western Sydney, by all means, with the reservoirs. But you can see there over space of time how those water resources are drying up. One of the things that the Blue Dot Observatory told us is that with this model of actually accessing the data on the common data repository, is they can process a month of data consisting of 7,000 bodies for only six euros. So it can be super cheap to do some serious science on these data sets. Another example, LiDAR data. So this is a US Geological Services has published a LiDAR point cloud consisting of 10 trillion points, as you might expect, it's over the US. But potentially it's actually got the ability to provide mechanisms for flood researchers, for agricultural researchers to do some serious science on very high resolution elevation data. So that's available on the open data repository as well. And one I'll just drill down to a little bit is called the NREL or the National Research Renewable Energy Laboratories, the Wind Integration National Data Set. Now what NREL have done is they've done a produced a weather model which has been corrected using observations, surface observations, scatterometer observations from satellites. And they produced a two kilometre resolution grid of the US. And with five minute temporal resolution spanning seven years at all the elevation, all the vertical elevations, the meaningful to commercial wind farm operators. That's 500 petabytes of HDF5 data. So it's a reasonably chunky set of data. Now what's actually built and that's published on the open data repository. And you can download that. If you've got a good pipe you can download that. But what most organisations are doing or what most consumers of this data, which are basically people who own their own wind farms, are doing is actually using a thing called the wind toolkit. And that's an open source set of tools that have basically been developed to specifically consume this data as well as reusing some other HDF5 formats. So there's a couple of components there. There's PIEWTK that provides an API for accessing the HDF5 data. And it's designed to be able to work in a download mode where you actually download these files. Or it can work in a native mode where you run it against loaded up as Lambda functions in AWS and directly query the S3 data. We see more and more of this as where open source projects will have an option to basically process data using cloud native services as well as downloading type services. There's a tool called WinVis which sits on top of PIEWTK. And another service I'll mention in a moment called HSDS. And that image there which should be moving but there's no player on this machine would show the wind barbs moving through time on that map. And that's a visualisation tool where you can zoom in on particular wind farm sites, different elevations and work out what the economic suitability of that site might be. There's another library which isn't actually specific to NREL but if you've got HDF5 data which there's an awful lot of that. It's the dominant format in meteorology. It's used by a lot of satellite formats. I imagine you've come across that as NETCDF or NETCDF4, that sort of thing. It's basically a data format that's optimised for large scale compute jobs. It more or less presents a database or a multi-dimensional data grid as a representative of a single file to a piece of code so it can basically just do a ray processing, dimensional ray processing directly on the file. So there's a library called H5pyd that's being designed to actually optimise the access to these big data sets. And the way it does that, and I'll just show you a little diagram here, is it can directly query the S3 objects. So you've got HDF5 data sitting in S3 as individual, probably multi-gigabyte files. And then it goes through a, does parallel queries against that data. And there's a little graphic there showing how that works. It basically chunks up the data sets, works out which blocks or which files or partitions in the data set it needs to retrieve, goes out and gets them all in parallel to S3 and then brings back the data set and presents it to the overlying application. So the benefit of that is that you can do potentially, these things can scale to, this library can scale to hundreds of nodes if need be working in parallel against S3. So they can very quickly process, allow algorithms to run directly against HDF5 data, even though the data is just sitting in place on S3, no downloads required. So if I just, there's a couple of use cases I've covered now. We've got a guide here, there's a URL which has also got a lot of the guidance about sharing and consuming data that I've mentioned. There's some additional use cases, Transport for London, the African Regional DataCube, Noah's NixRAD repository which is the composite radar, Doppler radar for the whole continental US, which we had here and that's shareable on the repository as well. So have a look at that guide, it's really useful resource. And lastly, this is usually the first slide in the deck but I hate leading with all sorts of cloudy economic type stuff but aside from what we've been talking about with cloud infrastructure, one thing to bear in mind is that some of the other key differences, and this applies to any cloud, not just AWS, is that it avoids that no upfront expense, you pay for what you use. Most of these science projects I've been talking about have cost literally 10, 15 cents, in some cases a couple of dollars to stand up and get running and they can be provision in minutes. So you only pay for what you use. Improves your market agility because you don't need to provision resources, sign contracts, you scale up or down and the whole thing self-service. Thank you. Thanks Craig, really good talk. So the nearest AWS region to here is Sydney, right? So that's another win for Australia, right? It sucked in New Zealand. So how many New Zealand organisations are part of the public data set program? How many organisations actually have a bucket sharing data? Dock would probably be the most proficient in that area. Dock? Dock, Department of Conservation, they've got ecological data sets that they are going through the onboarding process for this. MetService, we're talking to MetService about data as well, but Dock have also got a bunch of APIs for doing things like sites and national security. Are they part of the public data set program? No, they're actually conventional APIs. Cool, exciting. Questions from the audience? Matt, hold on. So, a lot of those examples that you used were from North America. What do you see as the biggest roadblock in this particular region, so Australia, New Zealand, Pacific Islands? It seems like different governments are deploying point solutions by different vendors. How do we bring this all together? Well, to answer your first question, the answer is, to be honest, it's funding. Those US organisations, the reason they do this is because they're funded to do this. They're funded to, Noah, for example, is funded to actually publish their weather models to the world. They've got a mandate to publish it to the world. Same with the Japanese meteorological organisation, they've got a mandate to do this. So, this just provides them with a channel to access the data. New Zealand, for various reasons and the way the organisations, most of the scientific organisations have been set up in New Zealand, they don't really have access to those sorts of funding resources and they've had to rely on commercialisation and a lot of their data. So, we're talking to them all the time about, let's see what we can actually share, but it's a long journey at the moment. The other question about fragmentation, the example I've shown is a US based, but there's a lot of European meteorological organisations, ESA, for example, the European Space Agency, they've been very prolific with the use of this repository as well. And actually, some of these Australian agencies are in the process of sharing data through this repository as well. So, there's nothing, we've actually got quite a few agencies talking to this, to our open data programme at the moment to try and get their data sets on board. Any other questions? I have heard that an agency, GS Ants Australia, may share some data through a cloud provider. Is that it? Cool. All right. Thank you very much. No worries.
 Hi everyone. So what I wanted to talk to you about first of all is just explain what is  the difference between running a geospatial workload on the cloud as opposed to on-premises  or any other type of infrastructure because you're probably aware that if you were running  a workload on any cloud, you could stand up a bunch of virtual machines and provision  a bunch of block storage and everything would look exactly the same as any other environment.  But there's one picture I'd like you to take away with you when you leave here and that's  that the cloud gives you the option of flipping that whole infrastructure on its head. So  instead of actually bringing the data to the compute infrastructure, you basically bring  the compute infrastructure to the data. And in the AWS environment, that data is represented  using S3 object stores. And the concept there is that S3 is a highly durable, highly scalable  object storage layer that pretty much any AWS service will be able to integrate with natively.  And so I've got Amazon Athena there. You can run SQL queries directly against CSV files  or JSON files or Parquet files sitting on S3. You don't need to load them into a database.  You can run Apache Spark jobs on them. You can do map reduce type tasks on S3 directly  without moving it from S3 into a Hadoop file system. You can have Amazon EC2 instances  directly referencing the S3 data. And I'll show you an example a little bit later where  we've got a bunch of data, the NREL organization running HD5 data, sorry, HDF5 sitting on S3  and directly accessing it through EC2 instances, again without moving it to block storage.  So that's probably the key differences because you're basically bringing the compute to the  storage. It takes a lot of the plumbing and a lot of the hassle out of dealing with data.  So the key phrase there is share, don't copy. By sharing the data, by publishing the data  once from the producers and actually consuming it in number of times globally across your  choice of infrastructure, whether it's work spaces for running say a heavy duty visualization  software stack that needs multiple GPUs and 4K monitors and hundreds of gigabytes of RAM  you can run that on a virtual desktop. You can do EC2 instances, map reducers I mentioned.  You can do serverless compute with Lambda where you actually don't actually stand up  servers at all. You write a piece of Python code, you put in pandas or your choice of  geospatial libraries and directly reference the S3 data. Or you can do machine learning  training based on the S3 data as well. And there's a bunch of customers of AWS in New  Zealand, in Wellington in fact, doing a whole lot of machine learning work with large image  files and large personal identifiable information sets as well.  So the reason we, what I want to talk about specifically though is about open data. So  taking that a step further, once we're actually talking about sharing data and bringing the  compute, it makes it very attractive for organizations to be able to publish their data and share  it using that S3 layer. And what we've been hearing from customers like Met Service and  Digital Globes, it'd be really great if someone could actually take the problem of having  to manage big FTP servers or file servers or web servers for distributing gigabytes  or even terabytes of data a day away from us and us having to ensure availability. And  likewise, the consumers of that data are telling us that what they really want is to avoid  that hassle of having to do regular downloads. When I was at Met Service, that was a big  part of our job was actually downloading global meteorological files from ECMWF, from the  UK Met Office, from GFS, from Japanese Meteorological Agency, satellite data all over the infrastructure,  all over the network infrastructure. That was almost like our number one headache. Compared  to that, running the weather models was relatively simple, being a bit flippant, but it's a big  job. So what AWS has responded to is by saying, well, why don't you let us actually manage  that storage for you? And if the data is actually genuinely useful to a large number of people,  we'll take on the actual storage costs and the data access costs as well. So the only  thing you need to do is basically provide, as a producer, is provide the content.  So probably the biggest advantage is that it opens up your data as a producer to a larger  community of users. We have a saying in AWS that 99% of the smartest people in the world  do not work for AWS. They work for someone else. And that's probably true of your organizations  as well, maybe not as much. But it's certainly the case that most of the brains actually  don't work for you. And you can leverage that by publishing your data. And I'll go a little  bit further than that. It's actually a lot more than just dumping the data somewhere  on S3. But by publishing that data, you open it up to that community. It lowers the cost  of research as well, because if you're not actually having to worry about network pipes  and that sort of thing, if you're able to run Jupyter Notebooks or write your own Python  code directly against the S3 data, it means you're actually running projects in a matter  of minutes and you're not having to invest in infrastructure as well.  So I'll talk to you a little bit about our public data sets already in our registry.  These are some of the major providers of those data sets. As you imagine, NASA publishes  a lot of satellite imagery. You can see the Hubble Space Telescope data there if you're  interested in it as well. But there's some surprising organisations there which probably  aren't geospatial in nature, but are national institutes of health. They publish de-identified  patient records into this repository as well. There's financial data. There's a stack of  meteorological data sources ranging from radar, satellite imagery, global weather models,  local area models like Waf and that sort of thing.  Now in terms of actually how you go about finding it, if you go to the AWS, there's  a URL there, open data at AWS, you'll see there's a registry and you can basically just  put in a search and that will give you all the information, a link to the data set and  it'll have a full page about what's actually, just back up, a full page about how to access  that data and it's usually just a URL. But more importantly, there's robust documentation  that sits behind it. There's tutorials, there's Jupyter notebooks, there's GitHub links  where you can actually access some tools that will make it easy to consume the data as well.  So in terms of how to share data, one of the things we've learnt through this open data  program is that there's more than just dropping the data somewhere and saying go for your  lives and make use of this. There's a few things you need to do. The first thing is  non-technical considerations associated with sharing. Number one is actually trust. If  someone's going to invest time in building a pipeline or a data processing pipeline  against your data, they need to be able to trust your data and that means that if you've  got any sort of remote sensing network with actual mechanical components or cameras or  whatever, those things go wrong. So when they go wrong, you need to be able to tell people  that it's gone wrong so they can actually adapt to that. Landsat's a really good example  of that, that they actually had some sensors fail on a satellite a few years ago and they  actually went bent over backwards to make sure that everyone knew about that and everyone  was factoring in those sensor failures into their software stacks. We used to have that  with MetService as well with some of the polar orbiters that over time they would break down,  sensors would get irradiated and they would stop working or lose sensitivity. And as a  provider, they were obliged to, they told us that and we were able to incorporate that.  So that really builds trust. Second thing is actually exactly the same as software,  you need good documentation, you need use cases, you need tutorials, you need sample  code that people can actually get started with. And lastly, accessibility. Again, if  people are going to put a lot of effort into tooling, they want to know they can get that  data when they need it, particularly if they're doing operational processing. Some of these  meteorological agencies are publishing hourly observations on this repository. It's not  just static data, it's actually live data. Some data sets are being updated every minute.  So if you've got consumers or if you are a consumer and you're consuming that data,  you want to know that that data is always there and you can get to it. And it's not  going to grind to a halt when lots of people are downloading it. So just that little curve  there, it means that what that's suggesting is that as the trustworthiness level of documentation  and accessibility of your data resource improves, your user base basically starts to grow pretty  steeply. If the data is really useful, people will persist and they'll get onto it and they'll  figure it out somehow. But if you're really going to access a large user base and particularly  busy people, you need to invest a bit of time.  The second thing is technical considerations. And this is a little bit caranturative, but  you can overthink and as well as underthink your data representation formats. Underthink  is pretty obvious. It's when you just take a binary database dump or something like that,  drop it onto S3 and say go for your lives. That's no good to anyone. Some people will  access it, but it's going to be very difficult for them. You can also overthink it. If you  design a handcraft or really fine-tuned API for a specific use case, you might think you're  doing people a favour and for some of the user base you are, but you're also going to alienate  some people that actually sit in the middle that need something a little bit more flexible  than originally defined API, but don't want to go and read binary database dumps. So ideally  what you're doing is actually trying to cater for multiple use cases. By all means, build  your API, but also make the data available in a more raw format so that people can actually  download it and do their own manipulations and slice it and dice it in the ways they  make sense for them.  So some common use cases, you can also optimise your data for some common use cases. I'll  show you an example with Wind Farm modelling in a couple of minutes where the provider  has actually gone through a techno economic analysis of the weather data. So they converted  wind speeds into actual shaft power outputs for the turbines. So local agencies or power  authorities could make decisions about how useful it would be to put a side of wind turbine  in that location. So moving from the meteorological domain to the economic domain. Build a map  in an index. Don't just drop it into a, don't just have a directory and trust that people  will understand when you file naming conventions. Build a searchable index. Provide a cherry  picking mechanism. So if you've built an index and people are able to target a slice of data  or build a query that can select a subset, build a mechanism by which they can extract  that data using range type selections. Because they'll save them a lot of time, they'll save  them a lot of download time as well. And lastly, track usage. Measure how people are using  your data. So I'll just finish up with a couple of examples. So one of them is on a common  data repository. One of them is called the Blue Dot Observatory. And what that is is  it's a consortium that's set up to analyse the Sentinel-2 satellite observation data.  And their objective is actually to look at patterns of water consumption and actually  the depletion of water resources throughout the world, particularly in the developing  world. And there's a pretty scary map showing that could be Sydney. If anyone's from Sydney,  that could be Western Sydney, by all means, with the reservoirs. But you can see there  over space of time how those water resources are drying up. One of the things that the  Blue Dot Observatory told us is that with this model of actually accessing the data  on the common data repository, is they can process a month of data consisting of 7,000  bodies for only six euros. So it can be super cheap to do some serious science on these  data sets. Another example, LiDAR data. So this is a US Geological Services has published  a LiDAR point cloud consisting of 10 trillion points, as you might expect, it's over the  US. But potentially it's actually got the ability to provide mechanisms for flood researchers,  for agricultural researchers to do some serious science on very high resolution elevation  data. So that's available on the open data repository as well. And one I'll just drill  down to a little bit is called the NREL or the National Research Renewable Energy Laboratories,  the Wind Integration National Data Set. Now what NREL have done is they've done a produced  a weather model which has been corrected using observations, surface observations, scatterometer  observations from satellites. And they produced a two kilometre resolution grid of the US.  And with five minute temporal resolution spanning seven years at all the elevation, all the  vertical elevations, the meaningful to commercial wind farm operators. That's 500 petabytes  of HDF5 data. So it's a reasonably chunky set of data. Now what's actually built and  that's published on the open data repository. And you can download that. If you've got a  good pipe you can download that. But what most organisations are doing or what most consumers  of this data, which are basically people who own their own wind farms, are doing is actually  using a thing called the wind toolkit. And that's an open source set of tools that have  basically been developed to specifically consume this data as well as reusing some other HDF5  formats. So there's a couple of components there. There's PIEWTK that provides an API  for accessing the HDF5 data. And it's designed to be able to work in a download mode where  you actually download these files. Or it can work in a native mode where you run it against  loaded up as Lambda functions in AWS and directly query the S3 data. We see more and more of  this as where open source projects will have an option to basically process data using  cloud native services as well as downloading type services. There's a tool called WinVis  which sits on top of PIEWTK. And another service I'll mention in a moment called HSDS. And  that image there which should be moving but there's no player on this machine would show  the wind barbs moving through time on that map. And that's a visualisation tool where  you can zoom in on particular wind farm sites, different elevations and work out what the  economic suitability of that site might be. There's another library which isn't actually  specific to NREL but if you've got HDF5 data which there's an awful lot of that. It's the  dominant format in meteorology. It's used by a lot of satellite formats. I imagine you've  come across that as NETCDF or NETCDF4, that sort of thing. It's basically a data format  that's optimised for large scale compute jobs. It more or less presents a database or a multi-dimensional  data grid as a representative of a single file to a piece of code so it can basically  just do a ray processing, dimensional ray processing directly on the file. So there's  a library called H5pyd that's being designed to actually optimise the access to these big  data sets. And the way it does that, and I'll just show you a little diagram here, is it  can directly query the S3 objects. So you've got HDF5 data sitting in S3 as individual,  probably multi-gigabyte files. And then it goes through a, does parallel queries against  that data. And there's a little graphic there showing how that works. It basically chunks  up the data sets, works out which blocks or which files or partitions in the data set  it needs to retrieve, goes out and gets them all in parallel to S3 and then brings back  the data set and presents it to the overlying application. So the benefit of that is that  you can do potentially, these things can scale to, this library can scale to hundreds of  nodes if need be working in parallel against S3. So they can very quickly process, allow  algorithms to run directly against HDF5 data, even though the data is just sitting in place  on S3, no downloads required. So if I just, there's a couple of use cases I've covered  now. We've got a guide here, there's a URL which has also got a lot of the guidance about  sharing and consuming data that I've mentioned. There's some additional use cases, Transport  for London, the African Regional DataCube, Noah's NixRAD repository which is the composite  radar, Doppler radar for the whole continental US, which we had here and that's shareable  on the repository as well. So have a look at that guide, it's really useful resource.  And lastly, this is usually the first slide in the deck but I hate leading with all sorts  of cloudy economic type stuff but aside from what we've been talking about with cloud infrastructure,  one thing to bear in mind is that some of the other key differences, and this applies  to any cloud, not just AWS, is that it avoids that no upfront expense, you pay for what  you use. Most of these science projects I've been talking about have cost literally 10,  15 cents, in some cases a couple of dollars to stand up and get running and they can be  provision in minutes. So you only pay for what you use. Improves your market agility  because you don't need to provision resources, sign contracts, you scale up or down and the  whole thing self-service. Thank you.  Thanks Craig, really good talk. So the nearest AWS region to here is Sydney, right? So that's  another win for Australia, right? It sucked in New Zealand. So how many New Zealand organisations  are part of the public data set program? How many organisations actually have a bucket  sharing data? Dock would probably be the most proficient in that area. Dock? Dock, Department  of Conservation, they've got ecological data sets that they are going through the onboarding  process for this. MetService, we're talking to MetService about data as well, but Dock  have also got a bunch of APIs for doing things like sites and national security. Are they  part of the public data set program? No, they're actually conventional APIs. Cool, exciting.  Questions from the audience? Matt, hold on.  So, a lot of those examples that you used were from North America. What do you see as  the biggest roadblock in this particular region, so Australia, New Zealand, Pacific  Islands? It seems like different governments are deploying point solutions by different  vendors. How do we bring this all together? Well, to answer your first question, the answer  is, to be honest, it's funding. Those US organisations, the reason they do this is because they're  funded to do this. They're funded to, Noah, for example, is funded to actually publish  their weather models to the world. They've got a mandate to publish it to the world.  Same with the Japanese meteorological organisation, they've got a mandate to do this. So, this  just provides them with a channel to access the data. New Zealand, for various reasons  and the way the organisations, most of the scientific organisations have been set up  in New Zealand, they don't really have access to those sorts of funding resources and they've  had to rely on commercialisation and a lot of their data. So, we're talking to them  all the time about, let's see what we can actually share, but it's a long journey at  the moment. The other question about fragmentation, the example I've shown is a US based, but  there's a lot of European meteorological organisations, ESA, for example, the European  Space Agency, they've been very prolific with the use of this repository as well. And actually,  some of these Australian agencies are in the process of sharing data through this repository  as well. So, there's nothing, we've actually got quite a few agencies talking to this,  to our open data programme at the moment to try and get their data sets on board.  Any other questions? I have heard that an agency, GS Ants Australia, may share some data through  a cloud provider. Is that it? Cool. All right. Thank you very much.  No worries.
Craig_Roach_-_Running_Geospatial_Workloads_on_AWS_a0ZqyPnjYq8_0_47.png Craig_Roach_-_Running_Geospatial_Workloads_on_AWS_a0ZqyPnjYq8_48_119.png Craig_Roach_-_Running_Geospatial_Workloads_on_AWS_a0ZqyPnjYq8_120_2855.png Craig_Roach_-_Running_Geospatial_Workloads_on_AWS_a0ZqyPnjYq8_2856_4295.png Craig_Roach_-_Running_Geospatial_Workloads_on_AWS_a0ZqyPnjYq8_4296_6443.png Craig_Roach_-_Running_Geospatial_Workloads_on_AWS_a0ZqyPnjYq8_6444_7811.png Craig_Roach_-_Running_Geospatial_Workloads_on_AWS_a0ZqyPnjYq8_7812_8903.png Craig_Roach_-_Running_Geospatial_Workloads_on_AWS_a0ZqyPnjYq8_8904_9311.png Craig_Roach_-_Running_Geospatial_Workloads_on_AWS_a0ZqyPnjYq8_9312_9827.png Craig_Roach_-_Running_Geospatial_Workloads_on_AWS_a0ZqyPnjYq8_9828_13295.png Craig_Roach_-_Running_Geospatial_Workloads_on_AWS_a0ZqyPnjYq8_13296_16535.png Craig_Roach_-_Running_Geospatial_Workloads_on_AWS_a0ZqyPnjYq8_16536_16619.png Craig_Roach_-_Running_Geospatial_Workloads_on_AWS_a0ZqyPnjYq8_16620_17255.png Craig_Roach_-_Running_Geospatial_Workloads_on_AWS_a0ZqyPnjYq8_17256_17615.png Craig_Roach_-_Running_Geospatial_Workloads_on_AWS_a0ZqyPnjYq8_17616_18155.png Craig_Roach_-_Running_Geospatial_Workloads_on_AWS_a0ZqyPnjYq8_18156_18911.png Craig_Roach_-_Running_Geospatial_Workloads_on_AWS_a0ZqyPnjYq8_18912_20687.png Craig_Roach_-_Running_Geospatial_Workloads_on_AWS_a0ZqyPnjYq8_20688_22403.png Craig_Roach_-_Running_Geospatial_Workloads_on_AWS_a0ZqyPnjYq8_22404_23675.png Craig_Roach_-_Running_Geospatial_Workloads_on_AWS_a0ZqyPnjYq8_23676_25043.png Craig_Roach_-_Running_Geospatial_Workloads_on_AWS_a0ZqyPnjYq8_25044_25943.png Craig_Roach_-_Running_Geospatial_Workloads_on_AWS_a0ZqyPnjYq8_25944_27035.png Craig_Roach_-_Running_Geospatial_Workloads_on_AWS_a0ZqyPnjYq8_27036_27143.png Craig_Roach_-_Running_Geospatial_Workloads_on_AWS_a0ZqyPnjYq8_27144_28031.png Craig_Roach_-_Running_Geospatial_Workloads_on_AWS_a0ZqyPnjYq8_28032_31187.png Craig_Roach_-_Running_Geospatial_Workloads_on_AWS_a0ZqyPnjYq8_31188_32339.png Craig_Roach_-_Running_Geospatial_Workloads_on_AWS_a0ZqyPnjYq8_32340_32342.png
Hi everyone. So what I wanted to talk to you about first of all is just explain what is the difference between running a geospatial workload on the cloud as opposed to on-premises or any other type of infrastructure because you're probably aware that if you were running a workload on any cloud, you could stand up a bunch of virtual machines and provision a bunch of block storage and everything would look exactly the same as any other environment. But there's one picture I'd like you to take away with you when you leave here and that's that the cloud gives you the option of flipping that whole infrastructure on its head. So instead of actually bringing the data to the compute infrastructure, you basically bring the compute infrastructure to the data. And in the AWS environment, that data is represented using S3 object stores. And the concept there is that S3 is a highly durable, highly scalable object storage layer that pretty much any AWS service will be able to integrate with natively. And so I've got Amazon Athena there. You can run SQL queries directly against CSV files or JSON files or Parquet files sitting on S3. You don't need to load them into a database. You can run Apache Spark jobs on them. You can do map reduce type tasks on S3 directly without moving it from S3 into a Hadoop file system. You can have Amazon EC2 instances directly referencing the S3 data. And I'll show you an example a little bit later where we've got a bunch of data, the NREL organization running HD5 data, sorry, HDF5 sitting on S3 and directly accessing it through EC2 instances, again without moving it to block storage. So that's probably the key differences because you're basically bringing the compute to the storage. It takes a lot of the plumbing and a lot of the hassle out of dealing with data. So the key phrase there is share, don't copy. By sharing the data, by publishing the data once from the producers and actually consuming it in number of times globally across your choice of infrastructure, whether it's work spaces for running say a heavy duty visualization software stack that needs multiple GPUs and 4K monitors and hundreds of gigabytes of RAM you can run that on a virtual desktop. You can do EC2 instances, map reducers I mentioned. You can do serverless compute with Lambda where you actually don't actually stand up servers at all. You write a piece of Python code, you put in pandas or your choice of geospatial libraries and directly reference the S3 data. Or you can do machine learning training based on the S3 data as well. And there's a bunch of customers of AWS in New Zealand, in Wellington in fact, doing a whole lot of machine learning work with large image files and large personal identifiable information sets as well. So the reason we, what I want to talk about specifically though is about open data. So taking that a step further, once we're actually talking about sharing data and bringing the compute, it makes it very attractive for organizations to be able to publish their data and share it using that S3 layer. And what we've been hearing from customers like Met Service and Digital Globes, it'd be really great if someone could actually take the problem of having to manage big FTP servers or file servers or web servers for distributing gigabytes or even terabytes of data a day away from us and us having to ensure availability. And likewise, the consumers of that data are telling us that what they really want is to avoid that hassle of having to do regular downloads. When I was at Met Service, that was a big part of our job was actually downloading global meteorological files from ECMWF, from the UK Met Office, from GFS, from Japanese Meteorological Agency, satellite data all over the infrastructure, all over the network infrastructure. That was almost like our number one headache. Compared to that, running the weather models was relatively simple, being a bit flippant, but it's a big job. So what AWS has responded to is by saying, well, why don't you let us actually manage that storage for you? And if the data is actually genuinely useful to a large number of people, we'll take on the actual storage costs and the data access costs as well. So the only thing you need to do is basically provide, as a producer, is provide the content. So probably the biggest advantage is that it opens up your data as a producer to a larger community of users. We have a saying in AWS that 99% of the smartest people in the world do not work for AWS. They work for someone else. And that's probably true of your organizations as well, maybe not as much. But it's certainly the case that most of the brains actually don't work for you. And you can leverage that by publishing your data. And I'll go a little bit further than that. It's actually a lot more than just dumping the data somewhere on S3. But by publishing that data, you open it up to that community. It lowers the cost of research as well, because if you're not actually having to worry about network pipes and that sort of thing, if you're able to run Jupyter Notebooks or write your own Python code directly against the S3 data, it means you're actually running projects in a matter of minutes and you're not having to invest in infrastructure as well. So I'll talk to you a little bit about our public data sets already in our registry. These are some of the major providers of those data sets. As you imagine, NASA publishes a lot of satellite imagery. You can see the Hubble Space Telescope data there if you're interested in it as well. But there's some surprising organisations there which probably aren't geospatial in nature, but are national institutes of health. They publish de-identified patient records into this repository as well. There's financial data. There's a stack of meteorological data sources ranging from radar, satellite imagery, global weather models, local area models like Waf and that sort of thing. Now in terms of actually how you go about finding it, if you go to the AWS, there's a URL there, open data at AWS, you'll see there's a registry and you can basically just put in a search and that will give you all the information, a link to the data set and it'll have a full page about what's actually, just back up, a full page about how to access that data and it's usually just a URL. But more importantly, there's robust documentation that sits behind it. There's tutorials, there's Jupyter notebooks, there's GitHub links where you can actually access some tools that will make it easy to consume the data as well. So in terms of how to share data, one of the things we've learnt through this open data program is that there's more than just dropping the data somewhere and saying go for your lives and make use of this. There's a few things you need to do. The first thing is non-technical considerations associated with sharing. Number one is actually trust. If someone's going to invest time in building a pipeline or a data processing pipeline against your data, they need to be able to trust your data and that means that if you've got any sort of remote sensing network with actual mechanical components or cameras or whatever, those things go wrong. So when they go wrong, you need to be able to tell people that it's gone wrong so they can actually adapt to that. Landsat's a really good example of that, that they actually had some sensors fail on a satellite a few years ago and they actually went bent over backwards to make sure that everyone knew about that and everyone was factoring in those sensor failures into their software stacks. We used to have that with MetService as well with some of the polar orbiters that over time they would break down, sensors would get irradiated and they would stop working or lose sensitivity. And as a provider, they were obliged to, they told us that and we were able to incorporate that. So that really builds trust. Second thing is actually exactly the same as software, you need good documentation, you need use cases, you need tutorials, you need sample code that people can actually get started with. And lastly, accessibility. Again, if people are going to put a lot of effort into tooling, they want to know they can get that data when they need it, particularly if they're doing operational processing. Some of these meteorological agencies are publishing hourly observations on this repository. It's not just static data, it's actually live data. Some data sets are being updated every minute. So if you've got consumers or if you are a consumer and you're consuming that data, you want to know that that data is always there and you can get to it. And it's not going to grind to a halt when lots of people are downloading it. So just that little curve there, it means that what that's suggesting is that as the trustworthiness level of documentation and accessibility of your data resource improves, your user base basically starts to grow pretty steeply. If the data is really useful, people will persist and they'll get onto it and they'll figure it out somehow. But if you're really going to access a large user base and particularly busy people, you need to invest a bit of time. The second thing is technical considerations. And this is a little bit caranturative, but you can overthink and as well as underthink your data representation formats. Underthink is pretty obvious. It's when you just take a binary database dump or something like that, drop it onto S3 and say go for your lives. That's no good to anyone. Some people will access it, but it's going to be very difficult for them. You can also overthink it. If you design a handcraft or really fine-tuned API for a specific use case, you might think you're doing people a favour and for some of the user base you are, but you're also going to alienate some people that actually sit in the middle that need something a little bit more flexible than originally defined API, but don't want to go and read binary database dumps. So ideally what you're doing is actually trying to cater for multiple use cases. By all means, build your API, but also make the data available in a more raw format so that people can actually download it and do their own manipulations and slice it and dice it in the ways they make sense for them. So some common use cases, you can also optimise your data for some common use cases. I'll show you an example with Wind Farm modelling in a couple of minutes where the provider has actually gone through a techno economic analysis of the weather data. So they converted wind speeds into actual shaft power outputs for the turbines. So local agencies or power authorities could make decisions about how useful it would be to put a side of wind turbine in that location. So moving from the meteorological domain to the economic domain. Build a map in an index. Don't just drop it into a, don't just have a directory and trust that people will understand when you file naming conventions. Build a searchable index. Provide a cherry picking mechanism. So if you've built an index and people are able to target a slice of data or build a query that can select a subset, build a mechanism by which they can extract that data using range type selections. Because they'll save them a lot of time, they'll save them a lot of download time as well. And lastly, track usage. Measure how people are using your data. So I'll just finish up with a couple of examples. So one of them is on a common data repository. One of them is called the Blue Dot Observatory. And what that is is it's a consortium that's set up to analyse the Sentinel-2 satellite observation data. And their objective is actually to look at patterns of water consumption and actually the depletion of water resources throughout the world, particularly in the developing world. And there's a pretty scary map showing that could be Sydney. If anyone's from Sydney, that could be Western Sydney, by all means, with the reservoirs. But you can see there over space of time how those water resources are drying up. One of the things that the Blue Dot Observatory told us is that with this model of actually accessing the data on the common data repository, is they can process a month of data consisting of 7,000 bodies for only six euros. So it can be super cheap to do some serious science on these data sets. Another example, LiDAR data. So this is a US Geological Services has published a LiDAR point cloud consisting of 10 trillion points, as you might expect, it's over the US. But potentially it's actually got the ability to provide mechanisms for flood researchers, for agricultural researchers to do some serious science on very high resolution elevation data. So that's available on the open data repository as well. And one I'll just drill down to a little bit is called the NREL or the National Research Renewable Energy Laboratories, the Wind Integration National Data Set. Now what NREL have done is they've done a produced a weather model which has been corrected using observations, surface observations, scatterometer observations from satellites. And they produced a two kilometre resolution grid of the US. And with five minute temporal resolution spanning seven years at all the elevation, all the vertical elevations, the meaningful to commercial wind farm operators. That's 500 petabytes of HDF5 data. So it's a reasonably chunky set of data. Now what's actually built and that's published on the open data repository. And you can download that. If you've got a good pipe you can download that. But what most organisations are doing or what most consumers of this data, which are basically people who own their own wind farms, are doing is actually using a thing called the wind toolkit. And that's an open source set of tools that have basically been developed to specifically consume this data as well as reusing some other HDF5 formats. So there's a couple of components there. There's PIEWTK that provides an API for accessing the HDF5 data. And it's designed to be able to work in a download mode where you actually download these files. Or it can work in a native mode where you run it against loaded up as Lambda functions in AWS and directly query the S3 data. We see more and more of this as where open source projects will have an option to basically process data using cloud native services as well as downloading type services. There's a tool called WinVis which sits on top of PIEWTK. And another service I'll mention in a moment called HSDS. And that image there which should be moving but there's no player on this machine would show the wind barbs moving through time on that map. And that's a visualisation tool where you can zoom in on particular wind farm sites, different elevations and work out what the economic suitability of that site might be. There's another library which isn't actually specific to NREL but if you've got HDF5 data which there's an awful lot of that. It's the dominant format in meteorology. It's used by a lot of satellite formats. I imagine you've come across that as NETCDF or NETCDF4, that sort of thing. It's basically a data format that's optimised for large scale compute jobs. It more or less presents a database or a multi-dimensional data grid as a representative of a single file to a piece of code so it can basically just do a ray processing, dimensional ray processing directly on the file. So there's a library called H5pyd that's being designed to actually optimise the access to these big data sets. And the way it does that, and I'll just show you a little diagram here, is it can directly query the S3 objects. So you've got HDF5 data sitting in S3 as individual, probably multi-gigabyte files. And then it goes through a, does parallel queries against that data. And there's a little graphic there showing how that works. It basically chunks up the data sets, works out which blocks or which files or partitions in the data set it needs to retrieve, goes out and gets them all in parallel to S3 and then brings back the data set and presents it to the overlying application. So the benefit of that is that you can do potentially, these things can scale to, this library can scale to hundreds of nodes if need be working in parallel against S3. So they can very quickly process, allow algorithms to run directly against HDF5 data, even though the data is just sitting in place on S3, no downloads required. So if I just, there's a couple of use cases I've covered now. We've got a guide here, there's a URL which has also got a lot of the guidance about sharing and consuming data that I've mentioned. There's some additional use cases, Transport for London, the African Regional DataCube, Noah's NixRAD repository which is the composite radar, Doppler radar for the whole continental US, which we had here and that's shareable on the repository as well. So have a look at that guide, it's really useful resource. And lastly, this is usually the first slide in the deck but I hate leading with all sorts of cloudy economic type stuff but aside from what we've been talking about with cloud infrastructure, one thing to bear in mind is that some of the other key differences, and this applies to any cloud, not just AWS, is that it avoids that no upfront expense, you pay for what you use. Most of these science projects I've been talking about have cost literally 10, 15 cents, in some cases a couple of dollars to stand up and get running and they can be provision in minutes. So you only pay for what you use. Improves your market agility because you don't need to provision resources, sign contracts, you scale up or down and the whole thing self-service. Thank you. Thanks Craig, really good talk. So the nearest AWS region to here is Sydney, right? So that's another win for Australia, right? It sucked in New Zealand. So how many New Zealand organisations are part of the public data set program? How many organisations actually have a bucket sharing data? Dock would probably be the most proficient in that area. Dock? Dock, Department of Conservation, they've got ecological data sets that they are going through the onboarding process for this. MetService, we're talking to MetService about data as well, but Dock have also got a bunch of APIs for doing things like sites and national security. Are they part of the public data set program? No, they're actually conventional APIs. Cool, exciting. Questions from the audience? Matt, hold on. So, a lot of those examples that you used were from North America. What do you see as the biggest roadblock in this particular region, so Australia, New Zealand, Pacific Islands? It seems like different governments are deploying point solutions by different vendors. How do we bring this all together? Well, to answer your first question, the answer is, to be honest, it's funding. Those US organisations, the reason they do this is because they're funded to do this. They're funded to, Noah, for example, is funded to actually publish their weather models to the world. They've got a mandate to publish it to the world. Same with the Japanese meteorological organisation, they've got a mandate to do this. So, this just provides them with a channel to access the data. New Zealand, for various reasons and the way the organisations, most of the scientific organisations have been set up in New Zealand, they don't really have access to those sorts of funding resources and they've had to rely on commercialisation and a lot of their data. So, we're talking to them all the time about, let's see what we can actually share, but it's a long journey at the moment. The other question about fragmentation, the example I've shown is a US based, but there's a lot of European meteorological organisations, ESA, for example, the European Space Agency, they've been very prolific with the use of this repository as well. And actually, some of these Australian agencies are in the process of sharing data through this repository as well. So, there's nothing, we've actually got quite a few agencies talking to this, to our open data programme at the moment to try and get their data sets on board. Any other questions? I have heard that an agency, GS Ants Australia, may share some data through a cloud provider. Is that it? Cool. All right. Thank you very much. No worries.
infrastructure to the data. And in the AWS sharing the data, by publishing the data once choice of geospatial libraries and directly Zealand, in Wellington in fact, doing a whole lot wind toolkit. And that's an open source set of resource. And lastly, this is usually the first from what we've been talking about with cloud access the data. New Zealand, for various reasons talking to this, to our open data programme at