State of GDAL
Rouault, Even
We will focus on recent developments and achievements in recent GDAL versions. In particular, new drivers such as FlatGeoBuf, Cloud Optimized GeoTIFF, EXR, HEIF, OGC API Tiles/Maps/Coverage, STAC Tiled Assets or the infrastructure to write vector drivers in Python. We will also present the multidimensional raster API and its tools. New utilities like gdal_viewshed will be introduced. The state and health of the community and its challenges will also be covered.
 Good day everyone and welcome to our morning session here in Cordoba. So we have a few interesting and exciting presentations from Evan and Angelos and Astrid and finally Ratclav on some popular projects. Our first presentation is from Evan Ruhl and he will speak to the state of GDAL and he will present the latest updates on the GDAL community as well as new features, drivers, tools to other new versions. Evan is the chair of the GDAL Project Steering Committee, the manager of Spatialist, which is a consultancy specialized in free and open source geospatial software development and he is obviously very familiar with GDAL, Proj, MapServer, QGIS and so on. Without further ado, I will move it over to Evan. Thank you Tom. So welcome everybody. My name is Evan Ruhl. I'm an independent free and open source software developer mostly focused on the GDAL, MapServer, Proj, DeepJotif and Q... I'm sorry, I have an issue with the sound going back to me. You might have to mute your venueless connection. Just mute the tab I think. Sorry for that. Okay it's better. Okay so I will give a quick update of what happened in the GDAL project during the last two years with 3.1 and I will talk a bit about the future direction. So what's GDAL in just one slide? So first GDAL stands for the Geospatial Data Abstraction Library. This is the black box you use often without realizing it if you want to read or write geospatial formats in most C++ open source or closed source GIS software. As of today it handles more than 250 different formats and as of recent years it also handles network protocol and services. It is delivered with an application programming interface for C++ and other languages such as Python, Java, C sharp. It comes with command line utilities to inspect file contents, perform file format conversion, image reproduction, rasterization, vectorization and many other operations. It chooses an MITX open source lightens which is super permissive and we release a version with new feature every six months and the BFIX releases every two months. So given that the last date of GDAL talk was two years ago I have now three releases to sum up. So in 3.1 we added a new dedicated driver to generate cloud optimized GeoT files. So typically you know just have to use a single GDAL translate evocation. There have been also a number of improvements in the internal layout of Cog files and the GeoT reader has been announced to reduce the number of HTTP GET requests needed in particular if you use a JPEG compression with binary transparency mask. If you're curious about the details of cloud optimized formats such as Cog you can see a later talk about that today given by Pyramid CalBear. A major work in this version was also the addition of a new API to read and write data sets that contain multidimensional arrays. So by multi-dimensional we mean something more than 2D such as XYZ or XYZ time. This is mostly of interest for formats such as NetCDF, HDF4, HDF5 or GRIB that are naturally multidimensional. Apart for this new API also being added to the in-memory and VRT drivers. So with a VRT driver you can create a virtual multidimensional array from different sources that can be then self-multidimensional or just classic 2D files such as GeoT. There are two new command line utilities, GDAL MDM Info to inspect the content and GDAL MDM Translate to convert or subset between formats. And in the future GDAL 3.4 version we will also have a new driver for that. I will speak about that a bit later. Other improvements, so we have new drivers like the E-AxRY, one for high dynamic range images. We have one for GeoEAT models and another one for the proprietary format by Regal. I'd like to draw attention on the flat GeoBuff vector driver which under reading and writing a new format which is optimized in particular for cloud access. It uses Hilbert R3s to enable fast bounding box spatial filtering and it also be mentioned in the talk I mentioned is a private slide. A map ML is a new candidate specification for W3C to have a standard inclusion of maps in hypertext documents. So this driver under the geometry part of the specification. GDAL warp has been improved to be able to directly create output in formats like PNG or JPEG that naturally don't support random writing which was a condition previously. And we have also GDAL ViewShade which is a new utility to compute ViewShade or Intervisibility. So here you have an example of that, it's an observer that is set on the Montmartre Sacré-Cœur Basilica in Paris and it uses the default settings of the utility. So in green you can see what the observer can see and in red what is masked. You can tune the observer position and 8, the observer is 8 and you can also tune the curvature coefficient which takes into account atmospheric refraction and so you might have to tune it depending on the wavelengths considered if it's invisible or radio. It is also now possible to write read-only vector drivers in Python. This can be used mostly for quick prototyping or conversion of two-storm or occasional file formats where we have a few examples, for example using CTJSUN specification. The OAPI driver which stands for OGC API feature has been updated to the 1.0 core specification. The GeoTiff driver has been improved to fix a long-standing performance issue when one created internal overviews on large files. It also has been updated to support the OGC GeoTiff 1.1 specification. For people who complain about cheffile being a multi-file and the issue that it could be, so the cheffile driver has been updated to be able to create and update ZIP to cheffiles. It works of course, you can experience some slowness because there is compression involved. For people liking NetCDF, we also now have written write support for the geometry part of the CF convention. GDAR 3.2 has received a new driver to support the not yet finalized OGC API for tiles, maps and coverages. It was based on the state of the specification and end of last year, so it might require some adjustment as it advances towards finalization. We have also a driver for the catch format that is used by S3 ArcGIS. We have a driver to read the HEIF and HEEC files. Dutch people will be happy to learn that we have no driver to read the cadastral vector format that is using the Netherlands. The new utility, GDAR Crate, has been added. It's a simple utility where you can just initialize the blank raster file from its size, its extent and the value to burn into it. Other improvements, so we have no multi-studied overview computation. We have almost speed up by a factor of 2 for the deflate compression. This requires a quite recent Liptif built against the LibDeflate open source project. The Cog driver can now generate tiles that are aligned with a well-known tiling scheme, such as the Google Mercator one. One can use a Cog file for example as a caching backend for a tile server, where each tile of the tiling scheme will correspond to a GeoTIF tile. The open file GeoData based driver has received a very welcome announcement with support for reading spatial indexes. So now it's really competitive to its appropriate equivalent for all reading operations. If you use bathymetric data sets or astronomical data, you'll benefit from improvements in the bugs and fits drivers. The NITF driver has also been improved to decode the specific metadata segment that are used in a new profile for multispectral and hyperspectral data sets. The vector feature model has been extended to support Unic constraint and alias properties which are used in a number of formats. And people still relying on the old way of importing the Python bindings will have to upgrade to the new way, which is actually 10 years old. So you have no excuse for not having a data get. In GDAL 3.3 we added the stack tag driver, which stands for spatial temporal catalog tag assets. This is an extension of the stack specification. There are several talks in the conference about stack, so I would like to watch them to get more details about stack. So this particular driver is a kind of WNTS in stack formalism. And you can use quite large metatiles, not just simple PNG or small JPEG files. So this can be very convenient to have both the advantage of cog and tiling. We have a new virtual file system for the Azure Data Lake storage Gen2 file system. We have also, you can also now store your GDAL configuration file in a ROSOS file, which will be loaded when GDAL initialized. We have enumerated and constrained and glob-filled domain support for the five geodab database drivers and geo package. And the Python utilities are now available in the GDAL utilities, Python package, and they can be used as Python collable functions. We have also, they procured and removed a few things. First, Python 2 support was dropped in favor of Python 3.6 and above. A few quite esoteric drivers have been completely moved away from the main repository to an auxiliary one, where they can be built as plugins if you really need them. And we have also marked a number of drivers for as they procreated and for removal in GDAL 3.5 unless we year for solid reasons to keep them. All of this is done in an effort to limit the continued growth of the code base and to be able to add new code for topics of more current interest. And to finish on the feature side of things, a small preview of what will be in GDAL 3.4, which will be released this November. So we will have a new driver, StackIT, which stands for StackItems, which use the projection extension specification. So we can build virtual mosaic for each image, which has published information regarding its projection and extent. And as I mentioned before, we will have also a new driver for ZAR, which will use a V2 specification, which is the one widely used and also for the ZAR V3 experimental specification. It supports both the classic 2D GDAL NPI and also the multi-dimensional one. And it will be optimized for multi-threaded decoding. Now going on more organizational topics, there's a big news item that you have probably heard about. So to make it short, we have set up a sports archive program to help funding maintenance activities. And maintenance here is to be understood in a broad term and encompasses all activities that are of general interest for the project, but are typically hard or impossible to fund through usual funding sources. As you may guess, GDAL is a large code base, about 1.5 million lines of code. We do have a regular flow of contribution by many people, but it's underlined by a relatively small pool of people that have a role of maintainer, which causes a sustainability problem. And GDAL is really used by a number of other geospatial software open source or proprietary. We have a page that lists more than 100 of them. So it's really a core foundation of geospatial technology and its well-being is quite critical. So we have approached NumFocus, which is a US charity to be our fiscal host. So NumFocus hosts a number of well-known projects such as NumPy, PANDAZ, SciPy, KandaForge, XRRA, Dask, and many others. And for other purposes, GDAL will remain a West Geo project. So this initiative was really successful as we have managed to raise about $300,000 per year, and many of the donors have done pledges for several years. So this will help secure funding for several regular commandeers and increase the best factor of the project. So currently we have two part-time commandeers, myself and Niel Dawson. We will be able to address activities such as bug fixing and triaging, timely review of pull requests, maintaining the continuous integration setups, and making the needed change to adopt for updates in the extreme dependency address security fixes and do release management. One topic that we will probably start soon thanks to this project is the addition of a CMake-based build system that will ultimately replace existing unix and Windows build system. And another good news is that with the sponsorship program we will be able to also sponsor enhancements in the upstream projects like PROD, GDOT, and FLEPTIF that are core foundations for GDAL. So I've put at the bottom of the slide the links to the two documents that explain the government rules of the sponsorship program. So if we have a look at your sponsors, you can see that we've managed to attract interest from three major cloud providers, two major satellite imagery providers, and a number of big and small and medium enterprises in the cheese industry. So many thanks to them for their support. And I also would particularly like to thank our Butler, Chris Holmes, and Paul Ramsey who have helped a lot setting up this initiative together with the support of the GDAL project steering committee. Thank you for your attention and I'm happy to answer any questions you may have. Great. Thank you, Evan. If anybody has any questions feel free to put them forth in the chat and we'll be sure to relay them. Maybe I'll start off with a question. How can – so it's exciting to hear about the sponsorship news and the sponsorship program. How can new sponsors approach the project? So on the GDAL.org website we have a new page that is dedicated to sponsorship and we have an email where people interested in sponsoring can contact us and we will make them in relationship with the staff at NumFocus that will take part of the details of setting up the sponsorship. That's great. And in terms of influence, what sort of value proposition, what kind of influence would the sponsors have on the project as a result of their contributions? That's a good question. So we have – so the sponsor, when they give to the project, is intended to be a non-directed way. That is, the GDAL project steering committee will remain responsible for deciding what will be done. However, we have put in place a structure where the sponsor will be able to explain about their use of GDAL, their ideas, and so we will take that as input for any decision on the project we may have. So basically the governments, the governance of the project will remain quite similar as it is today. We will continue to receive input and contribution for the whole community and sponsors will be able to give also their input and we'll try to make that as best as we can. Cool. Just looking to see if we have any questions from the audience. One more question on my side at least. You mentioned CMake which is encouraging. Is there a timeline on when this will land in Master? So first we will have to put together requests for comment document for approval and comment by the community. So the timeline I've in mind is the first capability in GDAL 3.5, so in MainExtra, which will be mostly for developers so they can start testing but probably not ready yet for production. In GDAL 3.6 it should be really close to be production ready and we will probably officially deprecate the existing file system and GDAL 3.7 will only keep CMake so that's my current idea of how it will go soon. A few more questions now. For the Cogdriver user wants to do cubic overviews on two bands but nearest on the third band using regenerate overviews works but starts emitting a warning about optimized layout broken. Is there any way to do this cleanly? Probably a detailed question but I'm not sure. Let me look at the question. Sorry, I must. Not really because if you go regenerate overviews it will indeed break the optimized layout of the file so I don't really have an easy solution for that. It's not super typical. Maybe the solution would be to create a regular JOT file using those different resampling methods and then use the option in the Cogdriver to use existing overviews to avoid them to be recomputed. That should work I think. Another question. How do you decide who will benefit from the sponsorship program? So for now it has been an invitation from the Gidal PSC. We will see how it works. It's really useful but probably at some points we will call for proposal to the community people have ideas that they want to be funded. We will probably have such a mechanism. Another question on the Cogdriver in relationship to JOTIF. What are your thoughts on when we should use plain old JOTIF as opposed to Cog? Cog is mostly plain old JOTIF so there is no real disadvantage in using Cog file. So it will work just the same. Yeah, Cog just makes sure that some things are put in the right place in the file for it to be in the most optimal layout for cloud usage but there is no real drawback in using a Cog file in a non-cloud context. Another question, when do you see Python 3 support being potentially dropped? Python 3.6? I don't know. I don't think there is any pressing reason for now to drop it. We haven't thought about that yet. Another question, does the line of sight prediction take into account any additional atmospheric refractivity information aside from just the wavelength of observation? I'm not sure I can answer the question. I was not the author of this utility. It was written by Thomas Secker. I just know there is this curvature coefficient that you can put as input to the utility. Okay, great. I think that's all of the questions. Evan, I'd like to thank you for always an interesting presentation. I'm glad to see the exciting news and developments in GDAL. And thank you very much for joining us here and have a good rest of the week. Thanks. Thanks. Great. Thank you.
 Good day everyone and welcome to our morning session here in Cordoba.  So we have a few interesting and exciting presentations from Evan and Angelos and Astrid  and finally Ratclav on some popular projects.  Our first presentation is from Evan Ruhl and he will speak to the state of GDAL and he  will present the latest updates on the GDAL community as well as new features, drivers,  tools to other new versions.  Evan is the chair of the GDAL Project Steering Committee, the manager of Spatialist, which  is a consultancy specialized in free and open source geospatial software development and  he is obviously very familiar with GDAL, Proj, MapServer, QGIS and so on.  Without further ado, I will move it over to Evan.  Thank you Tom.  So welcome everybody.  My name is Evan Ruhl.  I'm an independent free and open source software developer mostly focused on the GDAL, MapServer,  Proj, DeepJotif and Q...  I'm sorry, I have an issue with the sound going back to me.  You might have to mute your venueless connection.  Just mute the tab I think.  Sorry for that.  Okay it's better.  Okay so I will give a quick update of what happened in the GDAL project during the last  two years with 3.1 and I will talk a bit about the future direction.  So what's GDAL in just one slide?  So first GDAL stands for the Geospatial Data Abstraction Library.  This is the black box you use often without realizing it if you want to read or write  geospatial formats in most C++ open source or closed source GIS software.  As of today it handles more than 250 different formats and as of recent years it also handles  network protocol and services.  It is delivered with an application programming interface for C++ and other languages such  as Python, Java, C sharp.  It comes with command line utilities to inspect file contents, perform file format conversion,  image reproduction, rasterization, vectorization and many other operations.  It chooses an MITX open source lightens which is super permissive and we release a version  with new feature every six months and the BFIX releases every two months.  So given that the last date of GDAL talk was two years ago I have now three releases to  sum up.  So in 3.1 we added a new dedicated driver to generate cloud optimized GeoT files.  So typically you know just have to use a single GDAL translate evocation.  There have been also a number of improvements in the internal layout of Cog files and the  GeoT reader has been announced to reduce the number of HTTP GET requests needed in particular  if you use a JPEG compression with binary transparency mask.  If you're curious about the details of cloud optimized formats such as Cog you can see a  later talk about that today given by Pyramid CalBear.  A major work in this version was also the addition of a new API to read and write data  sets that contain multidimensional arrays.  So by multi-dimensional we mean something more than 2D such as XYZ or XYZ time.  This is mostly of interest for formats such as NetCDF, HDF4, HDF5 or GRIB that are naturally  multidimensional.  Apart for this new API also being added to the in-memory and VRT drivers.  So with a VRT driver you can create a virtual multidimensional array from different sources  that can be then self-multidimensional or just classic 2D files such as GeoT.  There are two new command line utilities, GDAL MDM Info to inspect the content and GDAL  MDM Translate to convert or subset between formats.  And in the future GDAL 3.4 version we will also have a new driver for that.  I will speak about that a bit later.  Other improvements, so we have new drivers like the E-AxRY, one for high dynamic range  images.  We have one for GeoEAT models and another one for the proprietary format by Regal.  I'd like to draw attention on the flat GeoBuff vector driver which under reading and writing  a new format which is optimized in particular for cloud access.  It uses Hilbert R3s to enable fast bounding box spatial filtering and it also be mentioned  in the talk I mentioned is a private slide.  A map ML is a new candidate specification for W3C to have a standard inclusion of maps  in hypertext documents.  So this driver under the geometry part of the specification.  GDAL warp has been improved to be able to directly create output in formats like PNG  or JPEG that naturally don't support random writing which was a condition previously.  And we have also GDAL ViewShade which is a new utility to compute ViewShade or Intervisibility.  So here you have an example of that, it's an observer that is set on the Montmartre  Sacré-Cœur Basilica in Paris and it uses the default settings of the utility.  So in green you can see what the observer can see and in red what is masked.  You can tune the observer position and 8, the observer is 8 and you can also tune the  curvature coefficient which takes into account atmospheric refraction and so you might have  to tune it depending on the wavelengths considered if it's invisible or radio.  It is also now possible to write read-only vector drivers in Python.  This can be used mostly for quick prototyping or conversion of two-storm or occasional file  formats where we have a few examples, for example using CTJSUN specification.  The OAPI driver which stands for OGC API feature has been updated to the 1.0 core specification.  The GeoTiff driver has been improved to fix a long-standing performance issue when one  created internal overviews on large files.  It also has been updated to support the OGC GeoTiff 1.1 specification.  For people who complain about cheffile being a multi-file and the issue that it could be,  so the cheffile driver has been updated to be able to create and update ZIP to cheffiles.  It works of course, you can experience some slowness because there is compression involved.  For people liking NetCDF, we also now have written write support for the geometry part  of the CF convention.  GDAR 3.2 has received a new driver to support the not yet finalized OGC API for tiles, maps  and coverages.  It was based on the state of the specification and end of last year, so it might require  some adjustment as it advances towards finalization.  We have also a driver for the catch format that is used by S3 ArcGIS.  We have a driver to read the HEIF and HEEC files.  Dutch people will be happy to learn that we have no driver to read the cadastral vector  format that is using the Netherlands.  The new utility, GDAR Crate, has been added.  It's a simple utility where you can just initialize the blank raster file from its size, its extent  and the value to burn into it.  Other improvements, so we have no multi-studied overview computation.  We have almost speed up by a factor of 2 for the deflate compression.  This requires a quite recent Liptif built against the LibDeflate open source project.  The Cog driver can now generate tiles that are aligned with a well-known tiling scheme,  such as the Google Mercator one.  One can use a Cog file for example as a caching backend for a tile server, where each tile  of the tiling scheme will correspond to a GeoTIF tile.  The open file GeoData based driver has received a very welcome announcement with support for  reading spatial indexes.  So now it's really competitive to its appropriate equivalent for all reading operations.  If you use bathymetric data sets or astronomical data, you'll benefit from improvements in  the bugs and fits drivers.  The NITF driver has also been improved to decode the specific metadata segment that  are used in a new profile for multispectral and hyperspectral data sets.  The vector feature model has been extended to support Unic constraint and alias properties  which are used in a number of formats.  And people still relying on the old way of importing the Python bindings will have to  upgrade to the new way, which is actually 10 years old.  So you have no excuse for not having a data get.  In GDAL 3.3 we added the stack tag driver, which stands for spatial temporal catalog  tag assets.  This is an extension of the stack specification.  There are several talks in the conference about stack, so I would like to watch them  to get more details about stack.  So this particular driver is a kind of WNTS in stack formalism.  And you can use quite large metatiles, not just simple PNG or small JPEG files.  So this can be very convenient to have both the advantage of cog and tiling.  We have a new virtual file system for the Azure Data Lake storage Gen2 file system.  We have also, you can also now store your GDAL configuration file in a ROSOS file, which  will be loaded when GDAL initialized.  We have enumerated and constrained and glob-filled domain support for the five geodab database  drivers and geo package.  And the Python utilities are now available in the GDAL utilities, Python package, and  they can be used as Python collable functions.  We have also, they procured and removed a few things.  First, Python 2 support was dropped in favor of Python 3.6 and above.  A few quite esoteric drivers have been completely moved away from the main repository to an  auxiliary one, where they can be built as plugins if you really need them.  And we have also marked a number of drivers for as they procreated and for removal in  GDAL 3.5 unless we year for solid reasons to keep them.  All of this is done in an effort to limit the continued growth of the code base and  to be able to add new code for topics of more current interest.  And to finish on the feature side of things, a small preview of what will be in GDAL 3.4,  which will be released this November.  So we will have a new driver, StackIT, which stands for StackItems, which use the projection  extension specification.  So we can build virtual mosaic for each image, which has published information regarding  its projection and extent.  And as I mentioned before, we will have also a new driver for ZAR, which will use a V2  specification, which is the one widely used and also for the ZAR V3 experimental specification.  It supports both the classic 2D GDAL NPI and also the multi-dimensional one.  And it will be optimized for multi-threaded decoding.  Now going on more organizational topics, there's a big news item that you have probably heard  about.  So to make it short, we have set up a sports archive program to help funding maintenance  activities.  And maintenance here is to be understood in a broad term and encompasses all activities  that are of general interest for the project, but are typically hard or impossible to fund  through usual funding sources.  As you may guess, GDAL is a large code base, about 1.5 million lines of code.  We do have a regular flow of contribution by many people, but it's underlined by a relatively  small pool of people that have a role of maintainer, which causes a sustainability problem.  And GDAL is really used by a number of other geospatial software open source or proprietary.  We have a page that lists more than 100 of them.  So it's really a core foundation of geospatial technology and its well-being is quite critical.  So we have approached NumFocus, which is a US charity to be our fiscal host.  So NumFocus hosts a number of well-known projects such as NumPy, PANDAZ, SciPy, KandaForge,  XRRA, Dask, and many others.  And for other purposes, GDAL will remain a West Geo project.  So this initiative was really successful as we have managed to raise about $300,000 per  year, and many of the donors have done pledges for several years.  So this will help secure funding for several regular commandeers and increase the best  factor of the project.  So currently we have two part-time commandeers, myself and Niel Dawson.  We will be able to address activities such as bug fixing and triaging, timely review  of pull requests, maintaining the continuous integration setups, and making the needed  change to adopt for updates in the extreme dependency address security fixes and do release  management.  One topic that we will probably start soon thanks to this project is the addition of  a CMake-based build system that will ultimately replace existing unix and Windows build system.  And another good news is that with the sponsorship program we will be able to also sponsor enhancements  in the upstream projects like PROD, GDOT, and FLEPTIF that are core foundations for GDAL.  So I've put at the bottom of the slide the links to the two documents that explain the  government rules of the sponsorship program.  So if we have a look at your sponsors, you can see that we've managed to attract interest  from three major cloud providers, two major satellite imagery providers, and a number  of big and small and medium enterprises in the cheese industry.  So many thanks to them for their support.  And I also would particularly like to thank our Butler, Chris Holmes, and Paul Ramsey  who have helped a lot setting up this initiative together with the support of the GDAL project  steering committee.  Thank you for your attention and I'm happy to answer any questions you may have.  Great.  Thank you, Evan.  If anybody has any questions feel free to put them forth in the chat and we'll be sure  to relay them.  Maybe I'll start off with a question.  How can – so it's exciting to hear about the sponsorship news and the sponsorship program.  How can new sponsors approach the project?  So on the GDAL.org website we have a new page that is dedicated to sponsorship and we have  an email where people interested in sponsoring can contact us and we will make them in relationship  with the staff at NumFocus that will take part of the details of setting up the sponsorship.  That's great.  And in terms of influence, what sort of value proposition, what kind of influence would  the sponsors have on the project as a result of their contributions?  That's a good question.  So we have – so the sponsor, when they give to the project, is intended to be a non-directed  way.  That is, the GDAL project steering committee will remain responsible for deciding what  will be done.  However, we have put in place a structure where the sponsor will be able to explain about  their use of GDAL, their ideas, and so we will take that as input for any decision on  the project we may have.  So basically the governments, the governance of the project will remain quite similar as  it is today.  We will continue to receive input and contribution for the whole community and sponsors will  be able to give also their input and we'll try to make that as best as we can.  Cool.  Just looking to see if we have any questions from the audience.  One more question on my side at least.  You mentioned CMake which is encouraging.  Is there a timeline on when this will land in Master?  So first we will have to put together requests for comment document for approval and comment  by the community.  So the timeline I've in mind is the first capability in GDAL 3.5, so in MainExtra, which  will be mostly for developers so they can start testing but probably not ready yet for  production.  In GDAL 3.6 it should be really close to be production ready and we will probably officially  deprecate the existing file system and GDAL 3.7 will only keep CMake so that's my current  idea of how it will go soon.  A few more questions now.  For the Cogdriver user wants to do cubic overviews on two bands but nearest on the third band  using regenerate overviews works but starts emitting a warning about optimized layout broken.  Is there any way to do this cleanly?  Probably a detailed question but I'm not sure.  Let me look at the question.  Sorry, I must.  Not really because if you go regenerate overviews it will indeed break the optimized layout of  the file so I don't really have an easy solution for that.  It's not super typical.  Maybe the solution would be to create a regular JOT file using those different resampling  methods and then use the option in the Cogdriver to use existing overviews to avoid them to  be recomputed.  That should work I think.  Another question.  How do you decide who will benefit from the sponsorship program?  So for now it has been an invitation from the Gidal PSC.  We will see how it works.  It's really useful but probably at some points we will call for proposal to the community  people have ideas that they want to be funded.  We will probably have such a mechanism.  Another question on the Cogdriver in relationship to JOTIF.  What are your thoughts on when we should use plain old JOTIF as opposed to Cog?  Cog is mostly plain old JOTIF so there is no real disadvantage in using Cog file.  So it will work just the same.  Yeah, Cog just makes sure that some things are put in the right place in the file for  it to be in the most optimal layout for cloud usage but there is no real drawback in using  a Cog file in a non-cloud context.  Another question, when do you see Python 3 support being potentially dropped?  Python 3.6?  I don't know.  I don't think there is any pressing reason for now to drop it.  We haven't thought about that yet.  Another question, does the line of sight prediction take into account any additional atmospheric  refractivity information aside from just the wavelength of observation?  I'm not sure I can answer the question.  I was not the author of this utility.  It was written by Thomas Secker.  I just know there is this curvature coefficient that you can put as input to the utility.  Okay, great.  I think that's all of the questions.  Evan, I'd like to thank you for always an interesting presentation.  I'm glad to see the exciting news and developments in GDAL.  And thank you very much for joining us here and have a good rest of the week.  Thanks.  Thanks.  Great.  Thank you.
FOSS4G_-_State_of_GDAL_iWxZSR1oE4A_0_131.png FOSS4G_-_State_of_GDAL_iWxZSR1oE4A_132_191.png FOSS4G_-_State_of_GDAL_iWxZSR1oE4A_192_2519.png FOSS4G_-_State_of_GDAL_iWxZSR1oE4A_2520_2615.png FOSS4G_-_State_of_GDAL_iWxZSR1oE4A_2616_2675.png FOSS4G_-_State_of_GDAL_iWxZSR1oE4A_2676_3023.png FOSS4G_-_State_of_GDAL_iWxZSR1oE4A_3024_3107.png FOSS4G_-_State_of_GDAL_iWxZSR1oE4A_3108_3191.png FOSS4G_-_State_of_GDAL_iWxZSR1oE4A_3192_3263.png FOSS4G_-_State_of_GDAL_iWxZSR1oE4A_3264_3407.png FOSS4G_-_State_of_GDAL_iWxZSR1oE4A_3408_3491.png FOSS4G_-_State_of_GDAL_iWxZSR1oE4A_3492_3539.png FOSS4G_-_State_of_GDAL_iWxZSR1oE4A_3540_4031.png FOSS4G_-_State_of_GDAL_iWxZSR1oE4A_4032_6047.png FOSS4G_-_State_of_GDAL_iWxZSR1oE4A_6048_7427.png FOSS4G_-_State_of_GDAL_iWxZSR1oE4A_7428_9407.png FOSS4G_-_State_of_GDAL_iWxZSR1oE4A_9408_11603.png FOSS4G_-_State_of_GDAL_iWxZSR1oE4A_11604_12611.png FOSS4G_-_State_of_GDAL_iWxZSR1oE4A_12612_13055.png FOSS4G_-_State_of_GDAL_iWxZSR1oE4A_13056_13151.png FOSS4G_-_State_of_GDAL_iWxZSR1oE4A_13152_13199.png FOSS4G_-_State_of_GDAL_iWxZSR1oE4A_13200_13247.png FOSS4G_-_State_of_GDAL_iWxZSR1oE4A_13248_13307.png FOSS4G_-_State_of_GDAL_iWxZSR1oE4A_13308_13355.png FOSS4G_-_State_of_GDAL_iWxZSR1oE4A_13356_13415.png FOSS4G_-_State_of_GDAL_iWxZSR1oE4A_13416_13463.png FOSS4G_-_State_of_GDAL_iWxZSR1oE4A_13464_14891.png FOSS4G_-_State_of_GDAL_iWxZSR1oE4A_14892_16535.png FOSS4G_-_State_of_GDAL_iWxZSR1oE4A_16536_16667.png FOSS4G_-_State_of_GDAL_iWxZSR1oE4A_16668_19871.png FOSS4G_-_State_of_GDAL_iWxZSR1oE4A_19872_22367.png FOSS4G_-_State_of_GDAL_iWxZSR1oE4A_22368_23771.png FOSS4G_-_State_of_GDAL_iWxZSR1oE4A_23772_25451.png FOSS4G_-_State_of_GDAL_iWxZSR1oE4A_25452_27851.png FOSS4G_-_State_of_GDAL_iWxZSR1oE4A_27852_30443.png FOSS4G_-_State_of_GDAL_iWxZSR1oE4A_30444_31331.png FOSS4G_-_State_of_GDAL_iWxZSR1oE4A_31332_31535.png FOSS4G_-_State_of_GDAL_iWxZSR1oE4A_31536_31607.png FOSS4G_-_State_of_GDAL_iWxZSR1oE4A_31608_33155.png FOSS4G_-_State_of_GDAL_iWxZSR1oE4A_33156_33203.png FOSS4G_-_State_of_GDAL_iWxZSR1oE4A_33204_40991.png FOSS4G_-_State_of_GDAL_iWxZSR1oE4A_40992_43991.png FOSS4G_-_State_of_GDAL_iWxZSR1oE4A_43992_44111.png
Good day everyone and welcome to our morning session here in Cordoba. So we have a few interesting and exciting presentations from Evan and Angelos and Astrid and finally Ratclav on some popular projects. Our first presentation is from Evan Ruhl and he will speak to the state of GDAL and he will present the latest updates on the GDAL community as well as new features, drivers, tools to other new versions. Evan is the chair of the GDAL Project Steering Committee, the manager of Spatialist, which is a consultancy specialized in free and open source geospatial software development and he is obviously very familiar with GDAL, Proj, MapServer, QGIS and so on. Without further ado, I will move it over to Evan. Thank you Tom. So welcome everybody. My name is Evan Ruhl. I'm an independent free and open source software developer mostly focused on the GDAL, MapServer, Proj, DeepJotif and Q... I'm sorry, I have an issue with the sound going back to me. You might have to mute your venueless connection. Just mute the tab I think. Sorry for that. Okay it's better. Okay so I will give a quick update of what happened in the GDAL project during the last two years with 3.1 and I will talk a bit about the future direction. So what's GDAL in just one slide? So first GDAL stands for the Geospatial Data Abstraction Library. This is the black box you use often without realizing it if you want to read or write geospatial formats in most C++ open source or closed source GIS software. As of today it handles more than 250 different formats and as of recent years it also handles network protocol and services. It is delivered with an application programming interface for C++ and other languages such as Python, Java, C sharp. It comes with command line utilities to inspect file contents, perform file format conversion, image reproduction, rasterization, vectorization and many other operations. It chooses an MITX open source lightens which is super permissive and we release a version with new feature every six months and the BFIX releases every two months. So given that the last date of GDAL talk was two years ago I have now three releases to sum up. So in 3.1 we added a new dedicated driver to generate cloud optimized GeoT files. So typically you know just have to use a single GDAL translate evocation. There have been also a number of improvements in the internal layout of Cog files and the GeoT reader has been announced to reduce the number of HTTP GET requests needed in particular if you use a JPEG compression with binary transparency mask. If you're curious about the details of cloud optimized formats such as Cog you can see a later talk about that today given by Pyramid CalBear. A major work in this version was also the addition of a new API to read and write data sets that contain multidimensional arrays. So by multi-dimensional we mean something more than 2D such as XYZ or XYZ time. This is mostly of interest for formats such as NetCDF, HDF4, HDF5 or GRIB that are naturally multidimensional. Apart for this new API also being added to the in-memory and VRT drivers. So with a VRT driver you can create a virtual multidimensional array from different sources that can be then self-multidimensional or just classic 2D files such as GeoT. There are two new command line utilities, GDAL MDM Info to inspect the content and GDAL MDM Translate to convert or subset between formats. And in the future GDAL 3.4 version we will also have a new driver for that. I will speak about that a bit later. Other improvements, so we have new drivers like the E-AxRY, one for high dynamic range images. We have one for GeoEAT models and another one for the proprietary format by Regal. I'd like to draw attention on the flat GeoBuff vector driver which under reading and writing a new format which is optimized in particular for cloud access. It uses Hilbert R3s to enable fast bounding box spatial filtering and it also be mentioned in the talk I mentioned is a private slide. A map ML is a new candidate specification for W3C to have a standard inclusion of maps in hypertext documents. So this driver under the geometry part of the specification. GDAL warp has been improved to be able to directly create output in formats like PNG or JPEG that naturally don't support random writing which was a condition previously. And we have also GDAL ViewShade which is a new utility to compute ViewShade or Intervisibility. So here you have an example of that, it's an observer that is set on the Montmartre Sacré-Cœur Basilica in Paris and it uses the default settings of the utility. So in green you can see what the observer can see and in red what is masked. You can tune the observer position and 8, the observer is 8 and you can also tune the curvature coefficient which takes into account atmospheric refraction and so you might have to tune it depending on the wavelengths considered if it's invisible or radio. It is also now possible to write read-only vector drivers in Python. This can be used mostly for quick prototyping or conversion of two-storm or occasional file formats where we have a few examples, for example using CTJSUN specification. The OAPI driver which stands for OGC API feature has been updated to the 1.0 core specification. The GeoTiff driver has been improved to fix a long-standing performance issue when one created internal overviews on large files. It also has been updated to support the OGC GeoTiff 1.1 specification. For people who complain about cheffile being a multi-file and the issue that it could be, so the cheffile driver has been updated to be able to create and update ZIP to cheffiles. It works of course, you can experience some slowness because there is compression involved. For people liking NetCDF, we also now have written write support for the geometry part of the CF convention. GDAR 3.2 has received a new driver to support the not yet finalized OGC API for tiles, maps and coverages. It was based on the state of the specification and end of last year, so it might require some adjustment as it advances towards finalization. We have also a driver for the catch format that is used by S3 ArcGIS. We have a driver to read the HEIF and HEEC files. Dutch people will be happy to learn that we have no driver to read the cadastral vector format that is using the Netherlands. The new utility, GDAR Crate, has been added. It's a simple utility where you can just initialize the blank raster file from its size, its extent and the value to burn into it. Other improvements, so we have no multi-studied overview computation. We have almost speed up by a factor of 2 for the deflate compression. This requires a quite recent Liptif built against the LibDeflate open source project. The Cog driver can now generate tiles that are aligned with a well-known tiling scheme, such as the Google Mercator one. One can use a Cog file for example as a caching backend for a tile server, where each tile of the tiling scheme will correspond to a GeoTIF tile. The open file GeoData based driver has received a very welcome announcement with support for reading spatial indexes. So now it's really competitive to its appropriate equivalent for all reading operations. If you use bathymetric data sets or astronomical data, you'll benefit from improvements in the bugs and fits drivers. The NITF driver has also been improved to decode the specific metadata segment that are used in a new profile for multispectral and hyperspectral data sets. The vector feature model has been extended to support Unic constraint and alias properties which are used in a number of formats. And people still relying on the old way of importing the Python bindings will have to upgrade to the new way, which is actually 10 years old. So you have no excuse for not having a data get. In GDAL 3.3 we added the stack tag driver, which stands for spatial temporal catalog tag assets. This is an extension of the stack specification. There are several talks in the conference about stack, so I would like to watch them to get more details about stack. So this particular driver is a kind of WNTS in stack formalism. And you can use quite large metatiles, not just simple PNG or small JPEG files. So this can be very convenient to have both the advantage of cog and tiling. We have a new virtual file system for the Azure Data Lake storage Gen2 file system. We have also, you can also now store your GDAL configuration file in a ROSOS file, which will be loaded when GDAL initialized. We have enumerated and constrained and glob-filled domain support for the five geodab database drivers and geo package. And the Python utilities are now available in the GDAL utilities, Python package, and they can be used as Python collable functions. We have also, they procured and removed a few things. First, Python 2 support was dropped in favor of Python 3.6 and above. A few quite esoteric drivers have been completely moved away from the main repository to an auxiliary one, where they can be built as plugins if you really need them. And we have also marked a number of drivers for as they procreated and for removal in GDAL 3.5 unless we year for solid reasons to keep them. All of this is done in an effort to limit the continued growth of the code base and to be able to add new code for topics of more current interest. And to finish on the feature side of things, a small preview of what will be in GDAL 3.4, which will be released this November. So we will have a new driver, StackIT, which stands for StackItems, which use the projection extension specification. So we can build virtual mosaic for each image, which has published information regarding its projection and extent. And as I mentioned before, we will have also a new driver for ZAR, which will use a V2 specification, which is the one widely used and also for the ZAR V3 experimental specification. It supports both the classic 2D GDAL NPI and also the multi-dimensional one. And it will be optimized for multi-threaded decoding. Now going on more organizational topics, there's a big news item that you have probably heard about. So to make it short, we have set up a sports archive program to help funding maintenance activities. And maintenance here is to be understood in a broad term and encompasses all activities that are of general interest for the project, but are typically hard or impossible to fund through usual funding sources. As you may guess, GDAL is a large code base, about 1.5 million lines of code. We do have a regular flow of contribution by many people, but it's underlined by a relatively small pool of people that have a role of maintainer, which causes a sustainability problem. And GDAL is really used by a number of other geospatial software open source or proprietary. We have a page that lists more than 100 of them. So it's really a core foundation of geospatial technology and its well-being is quite critical. So we have approached NumFocus, which is a US charity to be our fiscal host. So NumFocus hosts a number of well-known projects such as NumPy, PANDAZ, SciPy, KandaForge, XRRA, Dask, and many others. And for other purposes, GDAL will remain a West Geo project. So this initiative was really successful as we have managed to raise about $300,000 per year, and many of the donors have done pledges for several years. So this will help secure funding for several regular commandeers and increase the best factor of the project. So currently we have two part-time commandeers, myself and Niel Dawson. We will be able to address activities such as bug fixing and triaging, timely review of pull requests, maintaining the continuous integration setups, and making the needed change to adopt for updates in the extreme dependency address security fixes and do release management. One topic that we will probably start soon thanks to this project is the addition of a CMake-based build system that will ultimately replace existing unix and Windows build system. And another good news is that with the sponsorship program we will be able to also sponsor enhancements in the upstream projects like PROD, GDOT, and FLEPTIF that are core foundations for GDAL. So I've put at the bottom of the slide the links to the two documents that explain the government rules of the sponsorship program. So if we have a look at your sponsors, you can see that we've managed to attract interest from three major cloud providers, two major satellite imagery providers, and a number of big and small and medium enterprises in the cheese industry. So many thanks to them for their support. And I also would particularly like to thank our Butler, Chris Holmes, and Paul Ramsey who have helped a lot setting up this initiative together with the support of the GDAL project steering committee. Thank you for your attention and I'm happy to answer any questions you may have. Great. Thank you, Evan. If anybody has any questions feel free to put them forth in the chat and we'll be sure to relay them. Maybe I'll start off with a question. How can – so it's exciting to hear about the sponsorship news and the sponsorship program. How can new sponsors approach the project? So on the GDAL.org website we have a new page that is dedicated to sponsorship and we have an email where people interested in sponsoring can contact us and we will make them in relationship with the staff at NumFocus that will take part of the details of setting up the sponsorship. That's great. And in terms of influence, what sort of value proposition, what kind of influence would the sponsors have on the project as a result of their contributions? That's a good question. So we have – so the sponsor, when they give to the project, is intended to be a non-directed way. That is, the GDAL project steering committee will remain responsible for deciding what will be done. However, we have put in place a structure where the sponsor will be able to explain about their use of GDAL, their ideas, and so we will take that as input for any decision on the project we may have. So basically the governments, the governance of the project will remain quite similar as it is today. We will continue to receive input and contribution for the whole community and sponsors will be able to give also their input and we'll try to make that as best as we can. Cool. Just looking to see if we have any questions from the audience. One more question on my side at least. You mentioned CMake which is encouraging. Is there a timeline on when this will land in Master? So first we will have to put together requests for comment document for approval and comment by the community. So the timeline I've in mind is the first capability in GDAL 3.5, so in MainExtra, which will be mostly for developers so they can start testing but probably not ready yet for production. In GDAL 3.6 it should be really close to be production ready and we will probably officially deprecate the existing file system and GDAL 3.7 will only keep CMake so that's my current idea of how it will go soon. A few more questions now. For the Cogdriver user wants to do cubic overviews on two bands but nearest on the third band using regenerate overviews works but starts emitting a warning about optimized layout broken. Is there any way to do this cleanly? Probably a detailed question but I'm not sure. Let me look at the question. Sorry, I must. Not really because if you go regenerate overviews it will indeed break the optimized layout of the file so I don't really have an easy solution for that. It's not super typical. Maybe the solution would be to create a regular JOT file using those different resampling methods and then use the option in the Cogdriver to use existing overviews to avoid them to be recomputed. That should work I think. Another question. How do you decide who will benefit from the sponsorship program? So for now it has been an invitation from the Gidal PSC. We will see how it works. It's really useful but probably at some points we will call for proposal to the community people have ideas that they want to be funded. We will probably have such a mechanism. Another question on the Cogdriver in relationship to JOTIF. What are your thoughts on when we should use plain old JOTIF as opposed to Cog? Cog is mostly plain old JOTIF so there is no real disadvantage in using Cog file. So it will work just the same. Yeah, Cog just makes sure that some things are put in the right place in the file for it to be in the most optimal layout for cloud usage but there is no real drawback in using a Cog file in a non-cloud context. Another question, when do you see Python 3 support being potentially dropped? Python 3.6? I don't know. I don't think there is any pressing reason for now to drop it. We haven't thought about that yet. Another question, does the line of sight prediction take into account any additional atmospheric refractivity information aside from just the wavelength of observation? I'm not sure I can answer the question. I was not the author of this utility. It was written by Thomas Secker. I just know there is this curvature coefficient that you can put as input to the utility. Okay, great. I think that's all of the questions. Evan, I'd like to thank you for always an interesting presentation. I'm glad to see the exciting news and developments in GDAL. And thank you very much for joining us here and have a good rest of the week. Thanks. Thanks. Great. Thank you.
GDAL and he will present the latest updates on drivers, tools to other new versions. Evan is the such as XYZ or XYZ time. This is mostly of possible to write read-only vector drivers in The OAPI driver which stands for OGC API feature the state of the specification and end of last Python utilities are now available in the GDAL GDAL NPI and also the multi-dimensional one. And foundation of geospatial technology and its myself and Niel Dawson. We will be able to community and sponsors will be able to give also