v.in.lidar gives negative point count on large las files

[dnewcomb]

Trying to use v.in.lidar to import points from an las file with 3531181701 points

v.in.lidar -o -b input=/gisdata/grass7/lidardir/big_lidar_file_ground.laz output=whole_co_ground_pts Over-riding projection check Importing -763785595 points..

Seems to be importing so far..

Running GRASS 7.0 SVN snapshot from 10/25/14

[dnewcomb]

v.in.lidar has been running now for 10 hours and is using 13 GB of RAM for this instance. Write access to the sqlite database by other vector operations is currently blocked.

[mlennert]

Replying to dnewcomb:

Trying to use v.in.lidar to import points from an las file with 3531181701 points

v.in.lidar -o -b input=/gisdata/grass7/lidardir/big_lidar_file_ground.laz output=whole_co_ground_pts Over-riding projection check Importing -763785595 points..

A sign of integer overflow. AFAICT, in this particular case this is just in the message where a '%d' placeholder is used which represents a signed int. But the count of the number of points is defined as unsigned int and with larger files this will probably also overflow.

[mlennert]

Replying to dnewcomb:

v.in.lidar has been running now for 10 hours and is using 13 GB of RAM for this instance. Write access to the sqlite database by other vector operations is currently blocked.

Try importing with the -t flag to not create an attribute table. Or possibly use PostgreSQL as a back end for the attributes.

Moritz

[wenzeslaus]

Replying to mlennert:

Replying to dnewcomb:

Trying to use v.in.lidar to import points from an las file with 3531181701 points

Importing -763785595 points...

A sign of integer overflow. AFAICT, in this particular case this is just in the message where a '%d' placeholder is used which represents a signed int. But the count of the number of points is defined as unsigned int and with larger files this will probably also overflow.

Changed to unsigned long long with unsigned long as a fallback in r66255. Please test.

(This supposes usage without topology and attribute table.)

If the test works, please update the ​wiki and propose updates to the manual if needed.

[wenzeslaus]

There was one more issue with the formatting, two cases of unsigned long were still formatted as %d instead of %lu. Fixed in r66344. (There rest was already %lu and %llu accordingly.)

The info messages and percentage counts might be still wrong with large number of points as it is using what is obtained as point count from ​libLAS. This is the limitation of the library. It uses uint32_t in LASHeader_GetPointRecordsCount according to ​source code on GitHub and unsigned int according to the ​documentation. I made some changes to improve the situation and warn about the possible problem in r66345. The changes assume that libLAS will actually read the points correctly anyway and that the LAS format actually supports more than "4,294,967,296 - 1" points (32 bit unsigned integer). I did a quick look to libLAS but I made no conclusion.

Also, thanks to r66344 v.in.lidar now catches the case when we it runs out of categories as these are limited by GV_CAT_MAX (2,147,483,647) in dig_defines.h​:

#define PORT_INT_MAX    2147483647
#define GV_CAT_MAX   PORT_INT_MAX

Moreover, thanks to r66343 one can use return or class number as category value or not to use categories at all.

I wonder why the limit there is so low (for example, why not at least unsigned?), but on the other hand, it is an issue mainly for the point clouds where we are not interested in numbering individual features (using category as an ID) anyway.

[wenzeslaus]

This is fixed for v.in.lidar. Can somebody confirm that? v.info won't give the right number but that's a separate issue. Can we close this? No backport from 7.1/trunk to 7.0 branch is planned (this is a feature).

[wenzeslaus]

It was mostly counting and formatting issue and it should basically work in 7.0. Will be fixed in 7.2.0. Closing as fixed.