The PDAL Pointcloud Engine
Smith, Michael (US Army Corps of Engineers)
An introduction to the PDAL pointcloud library, how to accomplish basic data processing, read/write files and how to scale to do batch processing. Also covering the use of PDAL docker images for quick installation. Also covering various PDAL plugins, optional drivers and connections to other projects that use PDAL.
 Hello. Yeah. So welcome everyone to this session here. Where are we going to go into the clouds, but not up there where there's none, but into the point clouds, starting with Mike Smith, who's going to present PDAL. Just out of curiosity, how many people are using PDAL or Poodle or PDAL now? Okay. How many who haven't used it at all? Okay. Kind of an even number. So we'll kind of cover some advanced and cover some beginning stuff too. Just for those of you that are curious, this is a point cloud scan in the thermal range using a FLIR camera of Palmer Station in Antarctica. Just one of the places we've been able to scan. So a little bit about Poodle or PDAL, however you want to pronounce it. It's a BSD licensed library. So the most permissive available. We do support proprietary plug-ins. There are a number of proprietary plug-ins already, some of which are public and some which are not. It's a C++ library and we have a GitHub repo with pull requests. Gladly welcomed. We've had three official releases. The 1.1 release last year, just after Phosphor G Sol. We had a 1.2 release earlier this year and we're just going into a 1.3 beta release right now. So those of you that want to get out there and start testing things, find bugs, we welcome that. So the way Poodle is set up is very much like Git with a single Poodle command and then many sub-commands that run off of that. But Poodle is primarily a translation engine. You can see we have a large variety of readers, a large variety of writers. There's a reason the name Poodle is very similar to GDAL. It's very much inspired by GDAL. Howard Butler is a current committer on the GDAL project. So a lot of things are structured very similarly. It's intended to really be the GDAL for point clouds. And you can see we've just in the last release upcoming, we've added a bunch of new formats. Some of them scientific formats like Elvis and Icebridge that are primarily of interest to NASA. A new text reader that will help you get those XYZ text files into LAS, LAZ, something like that. And a couple of new writers, one of which I'll discuss. But the heart of Poodle is the filters. These are really the processing power of Poodle. A number of them are basic operations like filtering, splitting, chipping, sorting, things like that. Some of them are more noise processing like the Poisson filter, the statistical outlier. They're designed to really allow you to clean up your data. And then there are the main Poodle applications. These are kind of the top level commands. Some of these are basically wrappers around certain filters. For example, the Poodle ground command. This is just a way to get access to the filters.ground in a simpler manner. Some of them are very similar to GDAL. So there's Poodle translate just like there is GDAL translate. It's your basic command for changing file formats, doing minor changes to your files, other things like that. There's utility formats like Poodle T-index, I should say. Very similar to GDAL T-index. Building a tile list of all your files and allowing you to operate it on a single file once you've built that. And then there's Poodle pipeline, which is the main power of the Poodle processing engine. Here's what's really new in 1.3. JSON pipelines. XML is gone. Well, it's actually not gone. We'll preserve it around for two to three more releases. So XML is still supported, which is good because we built a lot of our applications around the XML pipelines. But we're in the process of converting over to JSON now. And the JSON pipelines make things much simpler. As we go through here, I'll show some examples of the Poodle pipeline in XML and then the complement in JSON. We have an enhanced derivative writer that allows you to write out things like slope, aspect, contours, hillshades. We have a bunch of new analysis filters that I briefly mentioned, the new T-index reader, which now gives you ability to do merge, clips, and filters right on a large number of files all with one command. Like I said, we have a better text reader now and improved argument validation, so much better feedback at the command line when you're filling out your arguments. And thanks to Connors work, transparent S3 URL handling. So if you store a lot of your data in S3, you can just reference an S3 URL in the command line. By the way, if people have questions, let me know during the talk. I'd rather do them when we have the context. So just give me your question and I'll repeat it for the audio, but we don't have to wait until the end for questions. Poodle ground, something that people like to do a lot with point clouds, classifying point cloud into ground and non-ground points. You can classify it, so write into the classification routines, or you can actually remove the points directly and make a smaller point cloud. Some LiDAR formats don't support classifications, so you actually have to remove the points. Things like LAS do support classifications, so you can just mark those points as ground. It uses an algorithm from the point cloud library that gets compiled into Poodle, the progressive morphological filter. And there is an approximate version, so when you're trying to play around with the parameters on Poodle ground, you might want to try with the approximate version because it's much faster to operate and then take approximate true once you get things narrowed down. So here's an example of a point cloud from Sitka, Alaska. It's, as you can see, quite heavily vegetated, and this was an area they wanted to produce a digital elevation model of. So we ran the Poodle ground filter on it, and now we have a bare earth point cloud. So this is the kinds of things you can do with the ground filter. Poodle Info is very similar to GDAL Info. It gives you, you can get basic summary information about your point clouds. There's several other options besides summary, things like just looking at the metadata, looking at the stats, as well as the ability, this is an overview of this area, the ability to get a boundary file for your point cloud. This is what's also run when you're generating a Poodle T-index, and it stores the boundary in the GDAL shapefile or whatever GDAL format you support. So by default, the parameters are fairly coarse. You get a very coarse outline for your data, but you can change the option to it, to the boundary filter, and get finer and finer boundary calculations at the cost of going through more data and doing more intensive calculations. One thing that's new is the boundary, the Poodle Info when you do a boundary, returns back the density because it's calculated the area and it knows the number of points. It will, however, vary based on the boundary that it calculated. So if you do a very coarse boundary, you might have a lower density than you do with a very fine boundary. Poodle Info can also give you information on specific points. You can drill down to your file and find are there bad points here, what are the specific ones, and then potentially remove those points. There's a split option that allows you to split very large point clouds into a series of tiles, either based on capacity, in this case, each of these point cloud tiles here is approximately three million points, or you can also specify in terms of length, and then you have approximately equal size tiles, depending on how much area you actually have, but they're going to vary in terms of point counts. Poodle T-index, very similar to GDAL T-index, allows you to run through a whole series of files and create a single OGR-compatible vector format that will store the path to all your files as well as a boundary for each file in the geometry column. Then this allows you to use this T-index for a variety of different operations, filtering, merging, clipping, without having to specify all the files that you put into it. You just reference the one single file. Here's an example where I took the split files, added a polygon geometry around it, ran this command, and I get the merged, clipped point cloud from the results of all those individual files. Poodle Translate is the main point cloud translation method, basically input outputs and then whatever different advanced features you want. The advanced options are very similar to the layer creation option and data set creation options that you see in GDAL. It assumes certain defaults which you can override whenever you want to or need to. Poodle Pipeline is basically the full power of Poodle. This allows you to put all your commands into one specific script and run it across one to many files. It allows you to run through the data just one time while doing multiple, if you need to, filter operations, multiple read operations, multiple write operations. In the end, most operations in Poodle go through a pipeline, whether it's transparent to you or not. Here's an example in the old XML format. Things start from the inside out, so they start with a reader, then go through a filter, this case a range filter, and filter between 0 and 99,999, and then in this case write out to a last file. Notice there's no file names in this particular pipeline. In this case, I'm specifying the file names on the command line, so I can keep this pipeline and just run through this and pass multiple file names. But XML is deprecated and now we're on to JSON. So you can see that the JSON format for specifying the pipeline is a lot simpler, a little bit less typing, definitely less angle brackets. And just like the colorization filter in XML, now in JSON, you can pass any GDAL compatible raster format to a point cloud. It does have to be in the same projection as the point cloud. And colorize the pipeline, basically it'll set the RGB values for those dimensions. Or you can just do it all at the command line, so there's multiple ways of doing the same thing. So here's an example of that particular raster, Sitka Alaska again, that I just pulled from the USGS. And then once you apply the colorization filter to your point cloud, you now have a colorized point cloud. All these images, by the way, were taken from cloud compare, which is another open source point cloud viewing engine that uses liblast to actually render things, doesn't use poodle yet, but that's forthcoming. Here's an example of using the reprojection filter, which is a common operation for point clouds. Again, this is the deprecated XML format, now in JSON. And you can specify your parameters at the command line, or you can even do a batch processing with XRs and do a whole bunch of operations. Another common thing to do with point clouds is generating digital elevation models. We have the points to grid output writer that we use that was provided to us from the open topography group. In this case, I've actually have the file name and the input and output file name in the point cloud, in the pipeline, which that's XML. Here it is in JSON. So running this pipeline command would run, read this file, filter it based on the classification. Classification 2 is ground, so it's going to cruise a bare earth digital elevation model, and then write it out at a one meter grid distance using in persist waiting output to a TIF file. Or here's the same kind of thing done with command line parameters rather than specifying in a pipeline. And you can even turn a series of command line parameters into a pipeline with the dash dash pipeline command. Other highlights about Poodle and the 1.3 release, we've added a new filters dot height that allows you to calculate normalized heights on a new dimension. One thing about this, though, it does have to be, it does have to have a ground classification, because otherwise we can't really tell where the base ground elevation is to calculate the normalized heights. We've added multiple thinning options. So we used to just have random, I believe it was, and now we've added Poisson and voxel grid thinning options. The filters dot attribute has been enhanced, so now you can use OGR features. So say you have a vector file of building footprints or other things like that, places you want to mask out, you can assign classifications from that vector format and apply it onto the point cloud data. And we've added a new density kernel command to look at density files. So here's an example of running the density command on that point cloud. These are just individual hex bins that are calculated fairly crude, but you can see that most of the points are captured along the hillside and down along the shore that really weren't too many points captured. Poodle now has a Python API available via PyPy. The ability to get your point cloud data into a NumPy array is as easy as four lines of Python now. Just open the pipeline, execute it, and then read it into the array. So very simple now to get your data into a NumPy array for analysis. We've refreshed the Poodle documentation. It's now in read-to-docs format with download options. The content has been completely reorganized and I think it's a lot simpler to read. Recently Howard gave a workshop at one of the NGOs and we have that on the website now, so you can download and work through your own workshop. It's 100 plus pages utilizing QGIS and Docker and goes through all the basic capabilities of Poodle. So you don't even have to pay for a workshop, you can just download it and run it. And we've added a bunch of new tutorials to the website. Poodle releases, source code will always be at poodle.io or clone the GitHub repo. The recommended way we have for people to get Poodle now is Docker. Docker is the fastest way to get Poodle. Just do Docker pull, Poodle, specify the release or leave it as latest and you're going to get an up-to-date runnable Poodle. We also have a dependencies Docker image that's really nice if you want to build your own Poodle image. So you want to compile in some custom argument, custom options, some different capabilities. All the dependencies are packaged together for you in a single Docker image and you can just use that as the base for your Docker. RPM is available, Debian unstable is available for Poodle. However, OSGO for W, we kind of need a champion. If there's any of you that want to work with Windows and want to build Poodle for us, contact us and let us know. And that's it. Okay, thank you, Mike, for this great overview of Poodle as you call it. Any questions? Yeah, Mike, does the Docker image contain all of the dependencies for all the optional things like the points-to-grid library and such? All the basic ones, yeah. There's some things that doesn't contain Oracle, stuff like that, but Postgres is there for PGPoint Cloud. Anything that's publicly available is available in the Poodle Docker image. There's some things that are non-distributable like the Oracle incident client, stuff like that that's not included in that. And one more question. The concerning calculating the density, the hex band. I tried to do that and it shows a visual representation, but can you do that to get what the point density is per meter square, let's say? The Poodle info command will give you the point density per meter squared overall for the whole file, not for individuals. So you can't make an image of the point density. You could because each one of those hex bins does have an area and it does have a count so you could calculate it yourself. Okay. All right, thank you. In the back. Okay. Thank you first. It looks amazing for me. This is just awesome. Maybe two quick questions for the conversion to DEM when I have a bit of sparse point cloud. Is there some form of interpolation or something like that? There is. I know currently they're being worked on as some newer improved digital elevation model calculations that are better at handling sparse data than the points to grid algorithm are. Those will probably be in the 1.3 release. They may not be fully documented at 1.3. They're kind of alpha level quality. So keep an eye on that and make comments in the GitHub tracker or other things like that where you find issues. The points to grid writer will do it, but it does have some issues sometimes. Okay. Okay. Thanks. And the second probably just shot in the dark, but when you talked about coloring point clouds, is there an inverse operation like from colored point cloud get the roster with the image actually? You could probably write out the XYZ RGB arrays and convert it from a, you know, basically you have to convert it back to a gridded format like the points to grid and then just colorize that from the XYZs. So there's nothing quick to do that, but you could do it manually. Thank you for the presentation. We've seen a boundary approximation. Is there something like a 3D representation of generalization of a? There isn't. It's 2D only currently. It's 2? 2D. Yeah. The boundary calculation. Will there be one? Yes. If you would like to submit a pull request. No, we could certainly take that under advisement and work on that. It's not something we currently have in the pipeline right now, but it certainly is something that could be added. Okay. Any other questions, remarks? I think then thanks a lot, Mike.
 Hello. Yeah. So welcome everyone to this session here. Where are we going to go into the clouds,  but not up there where there's none, but into the point clouds, starting with Mike Smith,  who's going to present PDAL.  Just out of curiosity, how many people are using PDAL or Poodle or PDAL now? Okay. How  many who haven't used it at all? Okay. Kind of an even number. So we'll kind of cover  some advanced and cover some beginning stuff too. Just for those of you that are curious,  this is a point cloud scan in the thermal range using a FLIR camera of Palmer Station  in Antarctica. Just one of the places we've been able to scan.  So a little bit about Poodle or PDAL, however you want to pronounce it. It's a BSD licensed  library. So the most permissive available. We do support proprietary plug-ins. There  are a number of proprietary plug-ins already, some of which are public and some which are  not. It's a C++ library and we have a GitHub repo with pull requests. Gladly welcomed.  We've had three official releases. The 1.1 release last year, just after Phosphor G Sol.  We had a 1.2 release earlier this year and we're just going into a 1.3 beta release right  now. So those of you that want to get out there and start testing things, find bugs,  we welcome that. So the way Poodle is set up is very much like  Git with a single Poodle command and then many sub-commands that run off of that. But  Poodle is primarily a translation engine. You can see we have a large variety of readers,  a large variety of writers. There's a reason the name Poodle is very similar to GDAL. It's  very much inspired by GDAL. Howard Butler is a current committer on the GDAL project.  So a lot of things are structured very similarly. It's intended to really be the GDAL for point  clouds. And you can see we've just in the last release  upcoming, we've added a bunch of new formats. Some of them scientific formats like Elvis  and Icebridge that are primarily of interest to NASA. A new text reader that will help  you get those XYZ text files into LAS, LAZ, something like that. And a couple of new  writers, one of which I'll discuss. But the heart of Poodle is the filters. These  are really the processing power of Poodle. A number of them are basic operations like  filtering, splitting, chipping, sorting, things like that. Some of them are more noise processing  like the Poisson filter, the statistical outlier. They're designed to really allow you to clean  up your data. And then there are the main Poodle applications. These are kind of the  top level commands. Some of these are basically wrappers around certain filters. For example,  the Poodle ground command. This is just a way to get access to the filters.ground in  a simpler manner. Some of them are very similar to GDAL. So there's Poodle translate just  like there is GDAL translate. It's your basic command for changing file formats, doing minor  changes to your files, other things like that. There's utility formats like Poodle  T-index, I should say. Very similar to GDAL T-index. Building a tile list of all your  files and allowing you to operate it on a single file once you've built that. And then  there's Poodle pipeline, which is the main power of the Poodle processing engine. Here's  what's really new in 1.3. JSON pipelines. XML is gone. Well, it's actually not gone.  We'll preserve it around for two to three more releases. So XML is still supported,  which is good because we built a lot of our applications around the XML pipelines. But  we're in the process of converting over to JSON now. And the JSON pipelines make things  much simpler. As we go through here, I'll show some examples of the Poodle pipeline  in XML and then the complement in JSON. We have an enhanced derivative writer that allows  you to write out things like slope, aspect, contours, hillshades. We have a bunch of new  analysis filters that I briefly mentioned, the new T-index reader, which now gives you  ability to do merge, clips, and filters right on a large number of files all with one command.  Like I said, we have a better text reader now and improved argument validation, so much  better feedback at the command line when you're filling out your arguments. And thanks to  Connors work, transparent S3 URL handling. So if you store a lot of your data in S3,  you can just reference an S3 URL in the command line. By the way, if people have questions,  let me know during the talk. I'd rather do them when we have the context. So just give  me your question and I'll repeat it for the audio, but we don't have to wait until the  end for questions. Poodle ground, something that people like to do a lot with point clouds,  classifying point cloud into ground and non-ground points. You can classify it, so write into  the classification routines, or you can actually remove the points directly and make a smaller  point cloud. Some LiDAR formats don't support classifications, so you actually have to remove  the points. Things like LAS do support classifications, so you can just mark those points as ground.  It uses an algorithm from the point cloud library that gets compiled into Poodle, the  progressive morphological filter. And there is an approximate version, so when you're  trying to play around with the parameters on Poodle ground, you might want to try with  the approximate version because it's much faster to operate and then take approximate  true once you get things narrowed down. So here's an example of a point cloud from Sitka,  Alaska. It's, as you can see, quite heavily vegetated, and this was an area they wanted  to produce a digital elevation model of. So we ran the Poodle ground filter on it, and  now we have a bare earth point cloud. So this is the kinds of things you can do with the  ground filter. Poodle Info is very similar to GDAL Info. It gives you, you can get basic  summary information about your point clouds. There's several other options besides summary,  things like just looking at the metadata, looking at the stats, as well as the ability,  this is an overview of this area, the ability to get a boundary file for your point cloud.  This is what's also run when you're generating a Poodle T-index, and it stores the boundary  in the GDAL shapefile or whatever GDAL format you support. So by default, the parameters  are fairly coarse. You get a very coarse outline for your data, but you can change the option  to it, to the boundary filter, and get finer and finer boundary calculations at the cost  of going through more data and doing more intensive calculations.  One thing that's new is the boundary, the Poodle Info when you do a boundary, returns  back the density because it's calculated the area and it knows the number of points. It  will, however, vary based on the boundary that it calculated. So if you do a very coarse  boundary, you might have a lower density than you do with a very fine boundary. Poodle Info  can also give you information on specific points. You can drill down to your file and  find are there bad points here, what are the specific ones, and then potentially remove  those points. There's a split option that allows you to split very large point clouds  into a series of tiles, either based on capacity, in this case, each of these point cloud tiles  here is approximately three million points, or you can also specify in terms of length,  and then you have approximately equal size tiles, depending on how much area you actually  have, but they're going to vary in terms of point counts.  Poodle T-index, very similar to GDAL T-index, allows you to run through a whole series of  files and create a single OGR-compatible vector format that will store the path to all your  files as well as a boundary for each file in the geometry column. Then this allows you  to use this T-index for a variety of different operations, filtering, merging, clipping, without  having to specify all the files that you put into it. You just reference the one single  file. Here's an example where I took the split files, added a polygon geometry around it,  ran this command, and I get the merged, clipped point cloud from the results of all those  individual files. Poodle Translate is the main point cloud translation method, basically  input outputs and then whatever different advanced features you want. The advanced options are  very similar to the layer creation option and data set creation options that you see  in GDAL. It assumes certain defaults which you can override whenever you want to or need  to. Poodle Pipeline is basically the full power of Poodle. This allows you to put all  your commands into one specific script and run it across one to many files. It allows  you to run through the data just one time while doing multiple, if you need to, filter  operations, multiple read operations, multiple write operations. In the end, most operations  in Poodle go through a pipeline, whether it's transparent to you or not. Here's an  example in the old XML format. Things start from the inside out, so they start with a  reader, then go through a filter, this case a range filter, and filter between 0 and 99,999,  and then in this case write out to a last file. Notice there's no file names in this  particular pipeline. In this case, I'm specifying the file names on the command line, so I can  keep this pipeline and just run through this and pass multiple file names. But XML is deprecated  and now we're on to JSON. So you can see that the JSON format for specifying the pipeline  is a lot simpler, a little bit less typing, definitely less angle brackets. And just like  the colorization filter in XML, now in JSON, you can pass any GDAL compatible raster format  to a point cloud. It does have to be in the same projection as the point cloud. And colorize  the pipeline, basically it'll set the RGB values for those dimensions. Or you can just  do it all at the command line, so there's multiple ways of doing the same thing. So  here's an example of that particular raster, Sitka Alaska again, that I just pulled from  the USGS. And then once you apply the colorization filter to your point cloud, you now have a  colorized point cloud. All these images, by the way, were taken from cloud compare, which  is another open source point cloud viewing engine that uses liblast to actually render  things, doesn't use poodle yet, but that's forthcoming. Here's an example of using the  reprojection filter, which is a common operation for point clouds. Again, this is the deprecated  XML format, now in JSON. And you can specify your parameters at the command line, or you  can even do a batch processing with XRs and do a whole bunch of operations. Another common  thing to do with point clouds is generating digital elevation models. We have the points  to grid output writer that we use that was provided to us from the open topography group.  In this case, I've actually have the file name and the input and output file name in  the point cloud, in the pipeline, which that's XML. Here it is in JSON. So running this pipeline  command would run, read this file, filter it based on the classification. Classification  2 is ground, so it's going to cruise a bare earth digital elevation model, and then write  it out at a one meter grid distance using in persist waiting output to a TIF file. Or  here's the same kind of thing done with command line parameters rather than specifying in  a pipeline. And you can even turn a series of command line parameters into a pipeline  with the dash dash pipeline command. Other highlights about Poodle and the 1.3 release,  we've added a new filters dot height that allows you to calculate normalized heights  on a new dimension. One thing about this, though, it does have to be, it does have to  have a ground classification, because otherwise we can't really tell where the base ground  elevation is to calculate the normalized heights. We've added multiple thinning options. So  we used to just have random, I believe it was, and now we've added Poisson and voxel  grid thinning options. The filters dot attribute has been enhanced, so now you can use OGR  features. So say you have a vector file of building footprints or other things like that,  places you want to mask out, you can assign classifications from that vector format and  apply it onto the point cloud data. And we've added a new density kernel command to look  at density files. So here's an example of running the density command on that point  cloud. These are just individual hex bins that are calculated fairly crude, but you  can see that most of the points are captured along the hillside and down along the shore  that really weren't too many points captured. Poodle now has a Python API available via  PyPy. The ability to get your point cloud data into a NumPy array is as easy as four  lines of Python now. Just open the pipeline, execute it, and then read it into the array.  So very simple now to get your data into a NumPy array for analysis. We've refreshed  the Poodle documentation. It's now in read-to-docs format with download options. The content  has been completely reorganized and I think it's a lot simpler to read. Recently Howard  gave a workshop at one of the NGOs and we have that on the website now, so you can download  and work through your own workshop. It's 100 plus pages utilizing QGIS and Docker and goes  through all the basic capabilities of Poodle. So you don't even have to pay for a workshop,  you can just download it and run it. And we've added a bunch of new tutorials to the website.  Poodle releases, source code will always be at poodle.io or clone the GitHub repo. The  recommended way we have for people to get Poodle now is Docker. Docker is the fastest  way to get Poodle. Just do Docker pull, Poodle, specify the release or leave it as latest  and you're going to get an up-to-date runnable Poodle. We also have a dependencies Docker  image that's really nice if you want to build your own Poodle image. So you want to compile  in some custom argument, custom options, some different capabilities. All the dependencies  are packaged together for you in a single Docker image and you can just use that as  the base for your Docker. RPM is available, Debian unstable is available for Poodle. However,  OSGO for W, we kind of need a champion. If there's any of you that want to work with  Windows and want to build Poodle for us, contact us and let us know. And that's it.  Okay, thank you, Mike, for this great overview of Poodle as you call it. Any questions?  Yeah, Mike, does the Docker image contain all of the dependencies for all the optional  things like the points-to-grid library and such?  All the basic ones, yeah. There's some things that doesn't contain Oracle, stuff like that,  but Postgres is there for PGPoint Cloud. Anything that's publicly available is available in  the Poodle Docker image. There's some things that are non-distributable like the Oracle  incident client, stuff like that that's not included in that.  And one more question. The concerning calculating the density, the hex band. I tried to do that  and it shows a visual representation, but can you do that to get what the point density  is per meter square, let's say?  The Poodle info command will give you the point density per meter squared overall for  the whole file, not for individuals.  So you can't make an image of the point density.  You could because each one of those hex bins does have an area and it does have a count  so you could calculate it yourself.  Okay.  All right, thank you.  In the back.  Okay. Thank you first. It looks amazing for me. This is just awesome. Maybe two quick  questions for the conversion to DEM when I have a bit of sparse point cloud. Is there  some form of interpolation or something like that?  There is. I know currently they're being worked on as some newer improved digital elevation  model calculations that are better at handling sparse data than the points to grid algorithm  are.  Those will probably be in the 1.3 release. They may not be fully documented at 1.3. They're  kind of alpha level quality.  So keep an eye on that and make comments in the GitHub tracker or other things like that  where you find issues.  The points to grid writer will do it, but it does have some issues sometimes.  Okay. Okay. Thanks.  And the second probably just shot in the dark, but when you talked about coloring point  clouds, is there an inverse operation like from colored point cloud get the roster with  the image actually?  You could probably write out the XYZ RGB arrays and convert it from a, you know, basically  you have to convert it back to a gridded format like the points to grid and then just colorize  that from the XYZs. So there's nothing quick to do that, but you could do it manually.  Thank you for the presentation. We've seen a boundary approximation. Is there something  like a 3D representation of generalization of a?  There isn't. It's 2D only currently.  It's 2?  2D. Yeah. The boundary calculation.  Will there be one?  Yes. If you would like to submit a pull request.  No, we could certainly take that under advisement and work on that. It's not something we currently  have in the pipeline right now, but it certainly is something that could be added.  Okay. Any other questions, remarks?  I think then thanks a lot, Mike.
foss4g-2016-1205-the_pdal_pointcloud_engine-hd_0_215.png foss4g-2016-1205-the_pdal_pointcloud_engine-hd_216_503.png foss4g-2016-1205-the_pdal_pointcloud_engine-hd_504_827.png foss4g-2016-1205-the_pdal_pointcloud_engine-hd_828_935.png foss4g-2016-1205-the_pdal_pointcloud_engine-hd_936_1211.png foss4g-2016-1205-the_pdal_pointcloud_engine-hd_1212_1583.png foss4g-2016-1205-the_pdal_pointcloud_engine-hd_1584_6623.png foss4g-2016-1205-the_pdal_pointcloud_engine-hd_6624_10475.png foss4g-2016-1205-the_pdal_pointcloud_engine-hd_10476_11111.png foss4g-2016-1205-the_pdal_pointcloud_engine-hd_11112_11543.png foss4g-2016-1205-the_pdal_pointcloud_engine-hd_11544_11603.png foss4g-2016-1205-the_pdal_pointcloud_engine-hd_11604_13439.png foss4g-2016-1205-the_pdal_pointcloud_engine-hd_13440_15071.png foss4g-2016-1205-the_pdal_pointcloud_engine-hd_15072_15503.png foss4g-2016-1205-the_pdal_pointcloud_engine-hd_15504_19487.png foss4g-2016-1205-the_pdal_pointcloud_engine-hd_19488_20627.png foss4g-2016-1205-the_pdal_pointcloud_engine-hd_20628_21839.png foss4g-2016-1205-the_pdal_pointcloud_engine-hd_21840_22283.png foss4g-2016-1205-the_pdal_pointcloud_engine-hd_22284_24347.png foss4g-2016-1205-the_pdal_pointcloud_engine-hd_24348_27851.png foss4g-2016-1205-the_pdal_pointcloud_engine-hd_27852_27959.png foss4g-2016-1205-the_pdal_pointcloud_engine-hd_27960_28007.png foss4g-2016-1205-the_pdal_pointcloud_engine-hd_28008_28055.png foss4g-2016-1205-the_pdal_pointcloud_engine-hd_28056_28187.png foss4g-2016-1205-the_pdal_pointcloud_engine-hd_28188_28247.png foss4g-2016-1205-the_pdal_pointcloud_engine-hd_28248_28319.png foss4g-2016-1205-the_pdal_pointcloud_engine-hd_28320_33179.png foss4g-2016-1205-the_pdal_pointcloud_engine-hd_33180_33263.png foss4g-2016-1205-the_pdal_pointcloud_engine-hd_33264_33421.png
Hello. Yeah. So welcome everyone to this session here. Where are we going to go into the clouds, but not up there where there's none, but into the point clouds, starting with Mike Smith, who's going to present PDAL. Just out of curiosity, how many people are using PDAL or Poodle or PDAL now? Okay. How many who haven't used it at all? Okay. Kind of an even number. So we'll kind of cover some advanced and cover some beginning stuff too. Just for those of you that are curious, this is a point cloud scan in the thermal range using a FLIR camera of Palmer Station in Antarctica. Just one of the places we've been able to scan. So a little bit about Poodle or PDAL, however you want to pronounce it. It's a BSD licensed library. So the most permissive available. We do support proprietary plug-ins. There are a number of proprietary plug-ins already, some of which are public and some which are not. It's a C++ library and we have a GitHub repo with pull requests. Gladly welcomed. We've had three official releases. The 1.1 release last year, just after Phosphor G Sol. We had a 1.2 release earlier this year and we're just going into a 1.3 beta release right now. So those of you that want to get out there and start testing things, find bugs, we welcome that. So the way Poodle is set up is very much like Git with a single Poodle command and then many sub-commands that run off of that. But Poodle is primarily a translation engine. You can see we have a large variety of readers, a large variety of writers. There's a reason the name Poodle is very similar to GDAL. It's very much inspired by GDAL. Howard Butler is a current committer on the GDAL project. So a lot of things are structured very similarly. It's intended to really be the GDAL for point clouds. And you can see we've just in the last release upcoming, we've added a bunch of new formats. Some of them scientific formats like Elvis and Icebridge that are primarily of interest to NASA. A new text reader that will help you get those XYZ text files into LAS, LAZ, something like that. And a couple of new writers, one of which I'll discuss. But the heart of Poodle is the filters. These are really the processing power of Poodle. A number of them are basic operations like filtering, splitting, chipping, sorting, things like that. Some of them are more noise processing like the Poisson filter, the statistical outlier. They're designed to really allow you to clean up your data. And then there are the main Poodle applications. These are kind of the top level commands. Some of these are basically wrappers around certain filters. For example, the Poodle ground command. This is just a way to get access to the filters.ground in a simpler manner. Some of them are very similar to GDAL. So there's Poodle translate just like there is GDAL translate. It's your basic command for changing file formats, doing minor changes to your files, other things like that. There's utility formats like Poodle T-index, I should say. Very similar to GDAL T-index. Building a tile list of all your files and allowing you to operate it on a single file once you've built that. And then there's Poodle pipeline, which is the main power of the Poodle processing engine. Here's what's really new in 1.3. JSON pipelines. XML is gone. Well, it's actually not gone. We'll preserve it around for two to three more releases. So XML is still supported, which is good because we built a lot of our applications around the XML pipelines. But we're in the process of converting over to JSON now. And the JSON pipelines make things much simpler. As we go through here, I'll show some examples of the Poodle pipeline in XML and then the complement in JSON. We have an enhanced derivative writer that allows you to write out things like slope, aspect, contours, hillshades. We have a bunch of new analysis filters that I briefly mentioned, the new T-index reader, which now gives you ability to do merge, clips, and filters right on a large number of files all with one command. Like I said, we have a better text reader now and improved argument validation, so much better feedback at the command line when you're filling out your arguments. And thanks to Connors work, transparent S3 URL handling. So if you store a lot of your data in S3, you can just reference an S3 URL in the command line. By the way, if people have questions, let me know during the talk. I'd rather do them when we have the context. So just give me your question and I'll repeat it for the audio, but we don't have to wait until the end for questions. Poodle ground, something that people like to do a lot with point clouds, classifying point cloud into ground and non-ground points. You can classify it, so write into the classification routines, or you can actually remove the points directly and make a smaller point cloud. Some LiDAR formats don't support classifications, so you actually have to remove the points. Things like LAS do support classifications, so you can just mark those points as ground. It uses an algorithm from the point cloud library that gets compiled into Poodle, the progressive morphological filter. And there is an approximate version, so when you're trying to play around with the parameters on Poodle ground, you might want to try with the approximate version because it's much faster to operate and then take approximate true once you get things narrowed down. So here's an example of a point cloud from Sitka, Alaska. It's, as you can see, quite heavily vegetated, and this was an area they wanted to produce a digital elevation model of. So we ran the Poodle ground filter on it, and now we have a bare earth point cloud. So this is the kinds of things you can do with the ground filter. Poodle Info is very similar to GDAL Info. It gives you, you can get basic summary information about your point clouds. There's several other options besides summary, things like just looking at the metadata, looking at the stats, as well as the ability, this is an overview of this area, the ability to get a boundary file for your point cloud. This is what's also run when you're generating a Poodle T-index, and it stores the boundary in the GDAL shapefile or whatever GDAL format you support. So by default, the parameters are fairly coarse. You get a very coarse outline for your data, but you can change the option to it, to the boundary filter, and get finer and finer boundary calculations at the cost of going through more data and doing more intensive calculations. One thing that's new is the boundary, the Poodle Info when you do a boundary, returns back the density because it's calculated the area and it knows the number of points. It will, however, vary based on the boundary that it calculated. So if you do a very coarse boundary, you might have a lower density than you do with a very fine boundary. Poodle Info can also give you information on specific points. You can drill down to your file and find are there bad points here, what are the specific ones, and then potentially remove those points. There's a split option that allows you to split very large point clouds into a series of tiles, either based on capacity, in this case, each of these point cloud tiles here is approximately three million points, or you can also specify in terms of length, and then you have approximately equal size tiles, depending on how much area you actually have, but they're going to vary in terms of point counts. Poodle T-index, very similar to GDAL T-index, allows you to run through a whole series of files and create a single OGR-compatible vector format that will store the path to all your files as well as a boundary for each file in the geometry column. Then this allows you to use this T-index for a variety of different operations, filtering, merging, clipping, without having to specify all the files that you put into it. You just reference the one single file. Here's an example where I took the split files, added a polygon geometry around it, ran this command, and I get the merged, clipped point cloud from the results of all those individual files. Poodle Translate is the main point cloud translation method, basically input outputs and then whatever different advanced features you want. The advanced options are very similar to the layer creation option and data set creation options that you see in GDAL. It assumes certain defaults which you can override whenever you want to or need to. Poodle Pipeline is basically the full power of Poodle. This allows you to put all your commands into one specific script and run it across one to many files. It allows you to run through the data just one time while doing multiple, if you need to, filter operations, multiple read operations, multiple write operations. In the end, most operations in Poodle go through a pipeline, whether it's transparent to you or not. Here's an example in the old XML format. Things start from the inside out, so they start with a reader, then go through a filter, this case a range filter, and filter between 0 and 99,999, and then in this case write out to a last file. Notice there's no file names in this particular pipeline. In this case, I'm specifying the file names on the command line, so I can keep this pipeline and just run through this and pass multiple file names. But XML is deprecated and now we're on to JSON. So you can see that the JSON format for specifying the pipeline is a lot simpler, a little bit less typing, definitely less angle brackets. And just like the colorization filter in XML, now in JSON, you can pass any GDAL compatible raster format to a point cloud. It does have to be in the same projection as the point cloud. And colorize the pipeline, basically it'll set the RGB values for those dimensions. Or you can just do it all at the command line, so there's multiple ways of doing the same thing. So here's an example of that particular raster, Sitka Alaska again, that I just pulled from the USGS. And then once you apply the colorization filter to your point cloud, you now have a colorized point cloud. All these images, by the way, were taken from cloud compare, which is another open source point cloud viewing engine that uses liblast to actually render things, doesn't use poodle yet, but that's forthcoming. Here's an example of using the reprojection filter, which is a common operation for point clouds. Again, this is the deprecated XML format, now in JSON. And you can specify your parameters at the command line, or you can even do a batch processing with XRs and do a whole bunch of operations. Another common thing to do with point clouds is generating digital elevation models. We have the points to grid output writer that we use that was provided to us from the open topography group. In this case, I've actually have the file name and the input and output file name in the point cloud, in the pipeline, which that's XML. Here it is in JSON. So running this pipeline command would run, read this file, filter it based on the classification. Classification 2 is ground, so it's going to cruise a bare earth digital elevation model, and then write it out at a one meter grid distance using in persist waiting output to a TIF file. Or here's the same kind of thing done with command line parameters rather than specifying in a pipeline. And you can even turn a series of command line parameters into a pipeline with the dash dash pipeline command. Other highlights about Poodle and the 1.3 release, we've added a new filters dot height that allows you to calculate normalized heights on a new dimension. One thing about this, though, it does have to be, it does have to have a ground classification, because otherwise we can't really tell where the base ground elevation is to calculate the normalized heights. We've added multiple thinning options. So we used to just have random, I believe it was, and now we've added Poisson and voxel grid thinning options. The filters dot attribute has been enhanced, so now you can use OGR features. So say you have a vector file of building footprints or other things like that, places you want to mask out, you can assign classifications from that vector format and apply it onto the point cloud data. And we've added a new density kernel command to look at density files. So here's an example of running the density command on that point cloud. These are just individual hex bins that are calculated fairly crude, but you can see that most of the points are captured along the hillside and down along the shore that really weren't too many points captured. Poodle now has a Python API available via PyPy. The ability to get your point cloud data into a NumPy array is as easy as four lines of Python now. Just open the pipeline, execute it, and then read it into the array. So very simple now to get your data into a NumPy array for analysis. We've refreshed the Poodle documentation. It's now in read-to-docs format with download options. The content has been completely reorganized and I think it's a lot simpler to read. Recently Howard gave a workshop at one of the NGOs and we have that on the website now, so you can download and work through your own workshop. It's 100 plus pages utilizing QGIS and Docker and goes through all the basic capabilities of Poodle. So you don't even have to pay for a workshop, you can just download it and run it. And we've added a bunch of new tutorials to the website. Poodle releases, source code will always be at poodle.io or clone the GitHub repo. The recommended way we have for people to get Poodle now is Docker. Docker is the fastest way to get Poodle. Just do Docker pull, Poodle, specify the release or leave it as latest and you're going to get an up-to-date runnable Poodle. We also have a dependencies Docker image that's really nice if you want to build your own Poodle image. So you want to compile in some custom argument, custom options, some different capabilities. All the dependencies are packaged together for you in a single Docker image and you can just use that as the base for your Docker. RPM is available, Debian unstable is available for Poodle. However, OSGO for W, we kind of need a champion. If there's any of you that want to work with Windows and want to build Poodle for us, contact us and let us know. And that's it. Okay, thank you, Mike, for this great overview of Poodle as you call it. Any questions? Yeah, Mike, does the Docker image contain all of the dependencies for all the optional things like the points-to-grid library and such? All the basic ones, yeah. There's some things that doesn't contain Oracle, stuff like that, but Postgres is there for PGPoint Cloud. Anything that's publicly available is available in the Poodle Docker image. There's some things that are non-distributable like the Oracle incident client, stuff like that that's not included in that. And one more question. The concerning calculating the density, the hex band. I tried to do that and it shows a visual representation, but can you do that to get what the point density is per meter square, let's say? The Poodle info command will give you the point density per meter squared overall for the whole file, not for individuals. So you can't make an image of the point density. You could because each one of those hex bins does have an area and it does have a count so you could calculate it yourself. Okay. All right, thank you. In the back. Okay. Thank you first. It looks amazing for me. This is just awesome. Maybe two quick questions for the conversion to DEM when I have a bit of sparse point cloud. Is there some form of interpolation or something like that? There is. I know currently they're being worked on as some newer improved digital elevation model calculations that are better at handling sparse data than the points to grid algorithm are. Those will probably be in the 1.3 release. They may not be fully documented at 1.3. They're kind of alpha level quality. So keep an eye on that and make comments in the GitHub tracker or other things like that where you find issues. The points to grid writer will do it, but it does have some issues sometimes. Okay. Okay. Thanks. And the second probably just shot in the dark, but when you talked about coloring point clouds, is there an inverse operation like from colored point cloud get the roster with the image actually? You could probably write out the XYZ RGB arrays and convert it from a, you know, basically you have to convert it back to a gridded format like the points to grid and then just colorize that from the XYZs. So there's nothing quick to do that, but you could do it manually. Thank you for the presentation. We've seen a boundary approximation. Is there something like a 3D representation of generalization of a? There isn't. It's 2D only currently. It's 2? 2D. Yeah. The boundary calculation. Will there be one? Yes. If you would like to submit a pull request. No, we could certainly take that under advisement and work on that. It's not something we currently have in the pipeline right now, but it certainly is something that could be added. Okay. Any other questions, remarks? I think then thanks a lot, Mike.
going to present PDAL. Just out of curiosity, how other things like that. There's utility formats to run through a whole series of files and create do a batch processing with XRs and do a whole dependencies for all the optional things like the points-to-grid library and such? All the basic the XYZs. So there's nothing quick to do that,