Opened 10 years ago
Closed 8 years ago
#2472 closed defect (fixed)
v.in.lidar gives negative point count on large las files
Reported by: | dnewcomb | Owned by: | wenzeslaus |
---|---|---|---|
Priority: | normal | Milestone: | 7.2.0 |
Component: | Default | Version: | svn-releasebranch70 |
Keywords: | v.in.lidar point initial count | Cc: | |
CPU: | x86-64 | Platform: | Unspecified |
Description
Trying to use v.in.lidar to import points from an las file with 3531181701 points
v.in.lidar -o -b input=/gisdata/grass7/lidardir/big_lidar_file_ground.laz output=whole_co_ground_pts Over-riding projection check Importing -763785595 points..
Seems to be importing so far..
Running GRASS 7.0 SVN snapshot from 10/25/14
Change History (8)
follow-up: 3 comment:1 by , 10 years ago
follow-up: 4 comment:2 by , 10 years ago
Replying to dnewcomb:
Trying to use v.in.lidar to import points from an las file with 3531181701 points
v.in.lidar -o -b input=/gisdata/grass7/lidardir/big_lidar_file_ground.laz output=whole_co_ground_pts Over-riding projection check Importing -763785595 points..
A sign of integer overflow. AFAICT, in this particular case this is just in the message where a '%d' placeholder is used which represents a signed int. But the count of the number of points is defined as unsigned int and with larger files this will probably also overflow.
comment:3 by , 10 years ago
Replying to dnewcomb:
v.in.lidar has been running now for 10 hours and is using 13 GB of RAM for this instance. Write access to the sqlite database by other vector operations is currently blocked.
Try importing with the -t flag to not create an attribute table. Or possibly use PostgreSQL as a back end for the attributes.
Moritz
comment:4 by , 9 years ago
Replying to mlennert:
Replying to dnewcomb:
Trying to use v.in.lidar to import points from an las file with 3531181701 points
Importing -763785595 points...
A sign of integer overflow. AFAICT, in this particular case this is just in the message where a '%d' placeholder is used which represents a signed int. But the count of the number of points is defined as unsigned int and with larger files this will probably also overflow.
Changed to unsigned long long
with unsigned long
as a fallback in r66255. Please test.
(This supposes usage without topology and attribute table.)
If the test works, please update the wiki and propose updates to the manual if needed.
comment:5 by , 9 years ago
There was one more issue with the formatting, two cases of unsigned long
were still formatted as %d
instead of %lu
. Fixed in r66344. (There rest was already %lu
and %llu
accordingly.)
The info messages and percentage counts might be still wrong with large number of points as it is using what is obtained as point count from libLAS. This is the limitation of the library. It uses uint32_t
in LASHeader_GetPointRecordsCount
according to source code on GitHub and unsigned int
according to the documentation. I made some changes to improve the situation and warn about the possible problem in r66345. The changes assume that libLAS will actually read the points correctly anyway and that the LAS format actually supports more than "4,294,967,296 - 1" points (32 bit unsigned integer). I did a quick look to libLAS but I made no conclusion.
Also, thanks to r66344 v.in.lidar now catches the case when we it runs out of categories as these are limited by GV_CAT_MAX
(2,147,483,647) in dig_defines.h:
#define PORT_INT_MAX 2147483647 #define GV_CAT_MAX PORT_INT_MAX
Moreover, thanks to r66343 one can use return or class number as category value or not to use categories at all.
I wonder why the limit there is so low (for example, why not at least unsigned?), but on the other hand, it is an issue mainly for the point clouds where we are not interested in numbering individual features (using category as an ID) anyway.
comment:6 by , 9 years ago
Owner: | changed from | to
---|
This is fixed for v.in.lidar. Can somebody confirm that? v.info won't give the right number but that's a separate issue. Can we close this? No backport from 7.1/trunk to 7.0 branch is planned (this is a feature).
comment:7 by , 8 years ago
Milestone: | 7.0.0 → 7.0.5 |
---|
comment:8 by , 8 years ago
Milestone: | 7.0.5 → 7.2.0 |
---|---|
Resolution: | → fixed |
Status: | new → closed |
It was mostly counting and formatting issue and it should basically work in 7.0. Will be fixed in 7.2.0. Closing as fixed.
v.in.lidar has been running now for 10 hours and is using 13 GB of RAM for this instance. Write access to the sqlite database by other vector operations is currently blocked.