GSoC 2017 Integration Of PDAL into GRASS GIS

Title:	Integration Of PDAL into GRASS GIS
Student Name:	Paul Schrum, ​North Carolina Sate University
Organization:	​OSGeo - Open Source Geospatial Foundation
Mentor Name:	​Vaclav Petras, ​Anna Petrasova
GSoC proposal:	​Integration of PDAL into GRASS GIS
Repository:	​v.in.pdal

Abstract

The Point Data Abstraction Library (PDAL) “is Point Data Abstraction Library, and it is an open source software for translating and processing point cloud data. It is not limited to just LiDAR data, although the focus and impetus for many of the tools have their origins in LiDAR..” At the time of this proposal, the only user interface for PDAL is a Command Line Interface. The proposal envisions the development of a GRASS GIS module which will facilitate importing and exporting Lidar files (with the .las extension) into and out of Vector and Raster formats.

Goal

The goal is to add API classes and methods to form an Adapter in GRASS GIS to utilize PDAL to read Lidar datasets. Included in this will be development of command line and GUI tools to make use of PDAL from GRASS easy for the end user operating through the GRASS application. This involves development of (or extension of) v.in.pdal, r.in.pdal, and r3.in.pdal and thorough unit test coverage of these. Optionally, if time permits, I will develop classes, methods, and UI elements to implement v.external and @PDAL pseudo-mapsets for point clouds.

Related Outstanding Tickets

​2732 (Enhancement): Read lidar data as vector points using PDAL

​3243: v.in.pdal build error: some PDAL 1.4.0 includes are not where the module expects them

Timeline

		status
Community bonding period	Get GRASS GIS and PDAL to build from source.
MAY 30 - JUNE 2	Begin development of v.in.pdal, including Unit Test coverage. Unit test development will be parallel to software development.
JUNE 5 - JUNE 9	Complete basic functionality of v.in.pdal at the command line. Begin development of GUI support. Basic functionality entails import and export of .las files to and from GRASS vector maps.
JUNE 12 - 16	Complete GUI development for v.in.pdal. Identify edge and corner cases, add them to the testing data, and handle them in code. Begin vacation 15 June.
JUNE 17 - 23	Vacation. Unavailable all week.
JUNE 24 - 30	Vacation until 26 June. Begin familiarization with GRASS's raster API. Complete First Evaluation.
First evaluations
JULY 1 - 7	Partial week due to holiday. Begin work on command line r.in.pdal. Plan Class design and determine adapter pattern specifics.
JULY 8 - 14	Continue development of r.in.pdal, including Unit Test coverage.
JULY 15 - 21	Fix bugs in v.in.pdal found by the mentor. Complete basic functionality of r.in.pdal and begin development of GUI support.
JULY 22 - 28	Complete GUI development of r.in.pdal. Identify edge and corner cases, add them to the testing data, and handle them in code. Complete Second Evaluation
Second evaluations
JULY 31 - AUG 4	Begin work on r3.in.pdal, including Class design and determining adapter pattern specifics.
AUGUST 7 - 11	Complete work on r3.in.pdal, including GUI and Unit Tests.
AUGUST 14 - 18	Complete all documentation and final checklist. Unavailable after 15 August due to the start of the Fall semester at NC State.
AUGUST 21 - 29	Complete final evaluation
Final evaluations

Requirements

​GRASS 7.3

Development

Detailed Proposed Task Plan

Week Ending	Task Descriptions
2 June	- Be able to debug GRASS using QCcreator. - Obtain suitable Lidar test data. Be able to step through and grasp code for v.in.lidar. - Be able to link with PDAL, call its functions, accept data that is returned. - Using Vashek's testing framework, have at least one unit test created and passing. - Set up sub-repository on github.

Weekly reports

Week 01

1. What did you get done this week?

• Built GRASS73 from source from my local repo directoy.
• Stepped into a module (v.in.ascii) using the debugger to watch it work at the source code level. This was only proving the concept of stepping into the code – just getting it to work.
• Talked with my mentor about the nature of the existing v.in.pdal module and what are some of the things which must be changed, fixed, or improved.

2. What do you plan on doing next week?

• Continue stepping through v.in.ascii to see how it works at the source level in order to understand how to write points to a vectormap.
• Get GRASS 73 to link to current version of PDAL.
• Fix broken references to PDAL data structures in the GRASS source
• Refactor existing v.in.pdal so that operations which r.in.pdal and r3.in.pdal will need are in a separate module and available as functions to be called from all three of them.
• Learn the process of calling PDAL and how to get it to do the basic operation of reading a lidar file and handing over the data to the caller.
• Set up a basic set of test data for PDAL, open it from pdal without GRASS, then open it using pdal through GRASS with only the most basic options in operation.
• Get PDAL to be able to read .laz (compressed) files. For some reason it can’t do that after build-from-source.

3. Are you blocked on anything?

I had two listed on the Wiki when I posted it, but my mentor helped me work through those before the end of the reporting period for Week 1. So there are currently no issues blocking me.

Week 02

1. What did you get done this week?

• Found a suitable test .las dataset and ran pdal on it from a unix command line just to view info and prove to myself that everything works as expected. I also ran v.in.lidar on this dataset, again just to be sure it works as expected.

• Got a good understanding of how writing to a new VectorMap works in GRASS by studying v.in.ascii and v.in.lidar.

• Added reference to pdal-config to my local build environment and got it to build okay. Doing this caused make to attempt to build v.in.pdal. See the next bullet item for more on this.

• I did not previously realize that when I was building GRASS from source, it was not trying to build v.in.pdal, though the source is part of the Subversion repo. So with my mentor’s help I brought v.in.pdal into the build process on my local machine. This triggered build errors, which is expected to happen. This is the state of the code as I prepare this report.

• Studied PDAL documentation (especially ​https://www.pdal.io/pipeline.html) and started to develop a coding approach and plan which will utilize Pipelines. There will be some common operations for which v.in.pdal will provide the pipeline json string internally so the GRASS end user does not have to concern themselves with it. There will also be a way in GRASS for end users to provide their own pipeline json file if they prefer. Much of this comes from recommendations by Howard Butler on the PDAL mailing list and in the primary GRASS enhancement ticket for this project.

• Discussed this approach with my mentor and we talked about a few more issues beyond just the pipeline approach.

• Added v.in.pdal to a new public github repository. (​https://github.com/PaulSchrum/v_in_pdal) As of 6/9/2017 this repository is just a copy of what is in the main svn repo for v.in.pdal. All changes will be pushed from my local machine to the github repository.

2. What do you plan on doing next week?

• Write classes and procedural code to make a simple and basic use case work via v.in.pdal.

• This includes implementing the pipeline approach as described above. It also entails eliminating rotted API function calls and struct references. Much of the existing code base will be rewritten. So part of this task will be assessing what existing code to retain and what to rework.

• Implement the GRASS testing framework using the sample dataset and add some foundational unit test runs.

• Paul’s vacation begins Friday, 16 June. That day and part of the Thursday before I will not be able to work on the project. Vacation lasts until Monday 26 June. Tuesday may be a slow day as well. I will continue to post weekly updates, but they will not have much in them.

3. Are you blocked on anything? No.

Week 03

1. What did you get done this week?

• I continued to gain an understanding of how similar GRASS commands work at a deep level so that when I modify the existing code for v.in.pdal, I will understand what I should do.

2. What do you plan on doing next week?

I will be on vacation from June 16 until June 27. My access to the internet may be limited during this time.

3. Are you blocked on anything?

No.

Week 04

1. What did you get done this week?

Nothing. I was on vacation all week as scheduled in my proposal.

2. What do you plan on doing next week?

Getting basic functionality working and one or more tests to confirm everything works correctly.

3. Are you blocked on anything?

No. My progress has been slower than expected, though, and if I still haven’t proceeded to the first of basic working code, then I will call it a block.

Week 05

1. What did you get done this week?

Achieved my goal of getting basic functionality to work. Specifically, the code now takes an input file name and an output file name via GRASS v.in.pdal and uses pdal to create the output file using a pipeline json string (created in memory only). At this point I have only tested this on pdal’s test file, 100-points.las from their github page. Further, I have only run the conversion to .txt format. But it is happening in GRASS code.

2. What do you plan on doing next week?

1. I need to develop code and a memory-only pipeline which will consume points from a .las file one point at a time so I can use them to build a GRASS Vector Map. In other words, get a basic, no-frills conversion from .las to GRASS to work like basic .las to .txt is now working.

2. Resolve the .o linking issue I am having with the added .cpp and .hpp files (see Blocked, below).

3. Get the process of testing/proving 2a into a unit test. Vashek has added a unit testing framework to GRASS under a previous GSoC project. I will be developing my tests under his established framework.

4. Add some common options to the GRASS command line option so we can do more than trivial conversions.

3. Are you blocked on anything?

I have added a new .cpp file and .hpp file for generating pipelines from GRASS inputs. They compile, but I can’t get them to link. I think I will need help figuring this out.

Week 06

1. What did you get done this week?

1. With my mentor's help, resolved the problem with the .o files not linking. Accomplishing this included adding the Makefile to the git repo. (It was not added originally because I inadvertently had Makefile listed in .gitignore, so I also removed that. ​Link to Commit)

2. Refactored function basicReaderWriter from dwelling in main.cpp to being found as a static function in the pipelinejson class. ​Commit

3. In consultation with my mentor (Vashek) determined to proceed by creating a new class, PdalVectorMapWriter, which will inherit from PDAL's Writer class. This will allow the GRASS Vector Map writer to be included as an end point in a json pipeline when invoked from GRASS. Implementing this has been started but not completed. This follows on to last weeks item 2a: "get a basic, no-frills conversion from .las to GRASS to work like basic .las to .txt is now working." ​Commit to add hpp and cpp files ​Commit refactoring existing files

2. What do you plan on doing next week?
3. Complete last week's item 2a, get a basic, no-frills conversion from .las to GRASS Vector Map to work.

2. Begin work on a basic conversion from .las to GRASS Raster. (r.in.pdal)

3. Complete last week's item 2c, Create unit tests for item 2a.

4. Begin work adding features to v.in.pdal.

3. Are you blocked on anything?

No.