Read lidar data as vector points using PDAL

[wenzeslaus]

​PDAL is a point cloud processing and format transformation library and supersedes ​libLAS (currently used in r.in.lidar and v.in.lidar) in many ways. Currently the most important difference is support for LAS 1.4. It also seems that we will have to replace libLAS by PDAL at some point.

I'm attaching a patch which is adding a module v.in.pointcloud. It is based on v.in.lidar and the difference is that it is using PDAL C++ API for reading LAS (so it can read LAS 1.4 except for waveform). LAZ is supported in the same way as with libLAS. When PDAL is compiled with LAZ, then v.in.pointcloud supports LAZ.

Serious issues with the patch:

• the compilation check in configure has just dummy code, I don't know how to make it compile C++

Potential improvements to patch:

• support for all PDAL supported (internally and by plugins) formats (should be easy to add), hence the name v.in.pointcloud
• documentation is just copied from v.in.lidar
• add test (test data are needed for that as well as for documentation)

Adding PDAL processing into the import or as separate module is clearly a possibility as well but this is out of scope of this particular patch.

General issues with PDAL for GRASS GIS:

• implemented in C++ and lacks C API (this is fixable)
• depends on C++11 (not a big issue when building modules)
• depends on Boost (hopefully will be removed in the future)
• potential limits on number of points as by default everything is ​in memory and limited to 2^32 - 1 points but there are ​ways around it

I need help with making the compilation check in configure compile C++, otherwise I think it can go trunk.

[wenzeslaus]

derived from v.in.lidar and using standard PDAL API with LasReader only

[wenzeslaus]

To try the patch you must ​compile PDAL which in short looks like:

git clone git@github.com:PDAL/PDAL.git pdal
cd pdal
mkdir build
cd build
cmake .. -DCMAKE_INSTALL_PREFIX:string=/pdal/install/dir

You may need to install some dependencies first (cmake will tell what).

Then apply the patch and add to your ./configure call for GRASS:

--with-pdal=yes --with-pdal-config=/pdal/install/dir/bin/pdal-config

It seems that PDAL is not generating pdal-config properly, until this gets solved, you need to find the file (/pdal/install/dir/bin/pdal-config) and add the following at the beginning (you may need to chmod u+w /pdal/install/dir/bin/pdal-config before editing the file):

INCLUDES="-I/pdal/install/dir/include/"
LIBS="-L/pdal/install/dir/lib/ -lpdalcpp -lpdal_util -lstdc++"

Unless you install system-wide and configure dynamic libraries, you need to do (for both GRASS compilation and running):

export LD_LIBRARY_PATH="/pdal/install/dir/lib:$LD_LIBRARY_PATH"

[wenzeslaus]

pdal-config is ​fixed now, so fix of INCLUDES and LIBS in the file /pdal/install/dir/bin/pdal-config is necessary.

To configure GRASS with PDAL, just compile PDAL and then run GRASS ./configure with:

--with-pdal=/home/vasek/dev/grass/pdal/pdal-bin/bin/pdal-config

Please note that the previous instructions said --with-pdal-config= but this is a wrong syntax.

LD_LIBRARY_PATH modification is always need for ./configure, compilation as well as running unless you do ldconfig as part of the installation (I'm not sure but for experimental purposes, the variable seems better, although less convenient).

[wenzeslaus]

First version now available in trunk. PDAL compilation support added in r67293 and v.in.pdal in r67436. Configure GRASS with:

./configure ... --with-pdal="/path/to/pdal-config"

PDAL can be compiled with or without LAZ (LAZlib) support and PCL support, so not all the functions may be available. If you compiled PDAL yourself you need to do also:

export LD_LIBRARY_PATH="/path/to/pdal/install/lib:$LD_LIBRARY_PATH"

I decided that the name v.in.pdal will be best because there are some PDAL-specific functions included. This includes also reprojection during import. However, the name can be changed, v.in.points sounds good as well.

The next steps are:

• add missing functions (implemented in GRASS) from libLAS-based v.in.lidar namely vector mask and count-based decimation
• add more ​PDAL-based filters (please suggest!)
• implement r.in.pdal similar to r.in.lidar (also needs list of filters to be implemented)
• implement r3.in.pdal
• implement v.out.pdal similar to v.out.lidar
• move some of the common functions to lib/lidar (most of the common functionality was already re-factored out from r.in.lidar and v.in.lidar and used in v.in.pdal and v.decimate

Things like mask or various decimations have (or can have) GRASS implementation and have PDAL implementation as well. It would make sense to do these operations at the beginning and at the end of the import process. If it would be at the beginning, only PDAL implementations can be used.

Note also that there are still some unresolved issues related to PDAL ability to handle large amounts of points although some solutions were proposed on the PDAL mailing list.

Finally, note also that PDAL is a processing library, so it can be also used in a separate module which would process vector points.

[neteler]

Replying to wenzeslaus:

• add more ​PDAL-based filters (please suggest!)

An idea could be to put an optional RGB point colorization here. It is a simple PDAL filter, hence v.in.pdal could read from an additionally specified RGB map retrieve the individual colors during import.

[wenzeslaus]

Replying to neteler:

Replying to wenzeslaus:

• add more ​PDAL-based filters (please suggest!)

An idea could be to put an optional RGB point colorization here. It is a simple PDAL filter, hence v.in.pdal could read from an additionally specified RGB map retrieve the individual colors during import.

Yes, this is actually on my list already. PDAL colorization could do a GeoTIFF while GRASS raster map would have to be done using native colorization implemented in GRASS. I think that this is exactly where ​using layers and categories for something else than ID and class will be necessary.

[Bas Couwenberg]

FYI: PDAL has been ​packaged for Debian and derivatives like Ubuntu.

[wenzeslaus]

Replying to sebastic:

FYI: PDAL has been ​packaged for Debian and derivatives like Ubuntu.

Thanks for the info. That's great. How do I install it? I can't find .deb, PPA or instructions at the ​official website.

[Bas Couwenberg]

Replying to wenzeslaus:

How do I install it? I can't find .deb, PPA or instructions at the ​official website.

Wait for the package to reach the mirrors, and then install it on a Debian unstable system with: apt-get install pdal libpdal-dev.

The former gets you the pdal commandline application, the latter the headers, libraries & pdal-config.

There is no PPA for Ubuntu, the package will be synced from Debian unstable into the Ubuntu release after xenial.

The PDAL website doesn't mention the Debian package because it is new and part of the distribution and not a 3rd party repository.

[neteler]

Milestone renamed

[wenzeslaus]

This is for the next release, although some code is already in place and backports are possible in this case.

[martinl]

Milestone renamed

[hobu]

There have been a number of PDAL developments since this ticket was first developed that might be interesting in relation to this task:

1) The PDAL ​PipelinExecutor class allows you to insert PDAL Pipeline JSON and get back PDAL PointViews quite easily. This means no more manual or laborious pdal::Stage construction. The executor class can hide all of that for you. Dig in the Python or Java extensions for inspiration on how to use it.

2) PDAL is now available via OSGeo4W64. We won't be committing to maintaining the 32-bit builds, but the 64-bit builds will be generated as part of PDAL's continuous integration and included in all major PDAL releases.

[ptschrum]

I (ptschrum) am working on this as a GSoC project, 2017.

[hellik]

citing a ​OSGeo4W ticket:

>no chance to fix the broken liblas/laszip stack?

There is a significant amount of software development to catch it up to the latest LASzip release. The LASzip API that libLAS used is no longer supported, and the new LASzip API is very different from the one that libLAS (and previously PDAL) used.

It would be better long term to get GRASS using PDAL. There was some effort on that topic, but I think it has stalled.

[hellik]

wiki of the ​GSoC 2017 project to integrate PDAL into GRASS.

[neteler]

Ticket retargeted after milestone closed

[wenzeslaus]

In 72246:

v.in.pdal: change API calls to PDAL 1.6.0, see #3496 and #2732 (author: felixg)

[wenzeslaus]

There is currently no owner of this issue.

[neteler]

In 72253:

v.in.pdal: change API calls to PDAL 1.6.0, see #3496 and #2732 (author: felixg, trunk r72246) + GPL header added in r72245

[wenzeslaus]

In 72363:

v.in.pdal: change API calls to PDAL 1.6.0, see #3496, #3243, #3101, #2732 (author: felixg, backport of trunk r72246) + GPL header added in r72245

[wenzeslaus]

To make this useful, we will need PDAL as a dependency in distributions. I don't know the procedure. Should I create ticket(s) for this here?

[hellik]

Replying to wenzeslaus:

To make this useful, we will need PDAL as a dependency in distributions. I don't know the procedure. Should I create ticket(s) for this here?

recent pdal versions (pdal-1.7.2) are only available in 64bit OSGeo4W.

32bit seems not to be updated for many years:

pdal-0.9.8-4.tar.bz2 05-Aug-2013 13:41 1.2M

[Bas Couwenberg]

You also need to ensure that LAZ support is optional, as LASzip has a license issue which makes it unredistributable, see:

• ​https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=750731#16
• ​https://trac.osgeo.org/osgeolive/ticket/1341#comment:2
• ​https://lists.osgeo.org/pipermail/standards/2015-April/000863.html

[martinl]

Replying to hellik:

recent pdal versions (pdal-1.7.2) are only available in 64bit OSGeo4W.

unfortunately pdal has a lot of dependencies:

eigen	(3.2.1-1)
	The Eigen library.
	Required by: pdal

hexer	(1.4.0-1)
	Hexer: GDAL-based hexagon density and boundary binning
	Required by: pdal

jq	(1.5.0-0)
	jq: command-line JSON processor
	Required by: pdal

jsoncpp	(1.6.0-1)
	JsonCpp library
	Required by: pdal

laszip	(3.2.2-1)
	The LASzip compression library
	Required by: pdal

laz-perf	(0.0.1-2)
	Alternative LAZ implementation for C/C++ and JavaScript
	Required by: pdal

nitro	(2.8.7-5)
	NITRO: C/C++ library for NITF manipulation
	Required by: pdal

oci	(12.1.0.1.0-1)
	Oracle Instant Client
	Required by: pdal

python3-core	(3.6.0-2)
	Python3 core interpreter and runtime
	Required by: python3-numpy, python3-gdal, python3-matplotlib, python3-pytz, python3-six, python3-cycler, python3-python-dateutil, python3-pyparsing

python3-cycler	(0.10.0-1)
	Composable style cycles
	Required by: python3-matplotlib

python3-gdal	(2.2.4-1)
	The GDAL/OGR Python3 Bindings and Scripts
	Required by: pdal

python3-matplotlib	(2.0.0-1)
	Python plotting package
	Required by: pdal

python3-numpy	(1.12.0+mkl-1)
	NumPy: array processing for numbers, strings, records, and objects.
	Required by: pdal, python3-gdal, python3-matplotlib

python3-pyparsing	(2.1.10-1)
	Python parsing module
	Required by: python3-matplotlib

python3-python-dateutil	(2.6.0-1)
	Extensions to the standard Python datetime module
	Required by: python3-matplotlib

python3-pytz	(2018.3-1)
	World timezone definitions, modern and historical
	Required by: python3-matplotlib

python3-six	(1.11.0-1)
	Python 2 and 3 compatibility utilities
	Required by: python3-matplotlib, python3-cycler, python3-python-dateutil

xz-devel	(5.2.3-1)
	XZ-format compression library - development files
	Required by: pdal

It will enlarge standalone binaries significantly (especially Python3). Would be nice to split OSGeo4W packages into pdal and python3-pdal.

[martinl]

See ​https://trac.osgeo.org/osgeo4w/ticket/576

[hobu]

You also need to ensure that LAZ support is optional, as LASzip has a license issue which makes it unredistributable, see:

Unredistributeable according to Debian's interpretations.

[wenzeslaus]

Replying to hobu:

You also need to ensure that LAZ support is optional, as LASzip has a license issue which makes it unredistributable, see:

Unredistributeable according to Debian's interpretations.

But there is las-perf, right? Are there some issues in using it and replacing LASzip completely?

[martinl]

Ticket retargeted after milestone closed

[martinl]

Ticket retargeted after milestone closed