Introduce a new 64-bit integer raster data type

[mlennert]

From #3084:

"Most importantly, a GRASS raster format for 64-bit signed integer. Note that GDAL does not support 64-bit signed or unsigned integers. The reason is probably that a portable implementation of 64-bit integers is not so easy. Regarding GRASS raster processing, the need for 64-bit integers usually arises only for raster maps with more than 2,147,483,647 cells which in turn usually requires Large File Support (LFS). Therefore the check for the availability of a 64-bit integer could be coupled to LFS. If support for 64-bit signed integer raster maps (say, LCELL) could be added to GRASS, users would need to stick to GRASS since GDAL raster export of 64-bit integers is not available. Interesting idea."

and

"As of r68944, i.segment supports regions with more than 2,147,483,647 cells, but the number of final objects must not exceed 2,147,483,647. This can only be solved by introducing a new 64 bit integer data type in addition to CELL, FCELL, DCELL, something for GRASS 8 maybe."

[mmetz]

Replying to mlennert:

From #3084:

"Most importantly, a GRASS raster format for 64-bit signed integer. Note that GDAL does not support 64-bit signed or unsigned integers. The reason is probably that a portable implementation of 64-bit integers is not so easy. Regarding GRASS raster processing, the need for 64-bit integers usually arises only for raster maps with more than 2,147,483,647 cells which in turn usually requires Large File Support (LFS). Therefore the check for the availability of a 64-bit integer could be coupled to LFS. If support for 64-bit signed integer raster maps (say, LCELL) could be added to GRASS, users would need to stick to GRASS since GDAL raster export of 64-bit integers is not available. Interesting idea."

A 64 bit signed integer type has been added to trunk with r72230. Various other projects, e.g. GDAL, SQLite, HDF5, Python also have some 64 bit signed integer type. In order to avoid namespace collision, the new type has been defined as grass_int64 and LCELL. GRASS libraries and modules do not yet make use of this type. Defining a 64 bit signed integer type has been tested on Linux + gcc, FreeBSD + clang, and MS Windows + gcc (mingw).

[mlennert]

Replying to mmetz:

Replying to mlennert:

From #3084:

"Most importantly, a GRASS raster format for 64-bit signed integer. Note that GDAL does not support 64-bit signed or unsigned integers. The reason is probably that a portable implementation of 64-bit integers is not so easy. Regarding GRASS raster processing, the need for 64-bit integers usually arises only for raster maps with more than 2,147,483,647 cells which in turn usually requires Large File Support (LFS). Therefore the check for the availability of a 64-bit integer could be coupled to LFS. If support for 64-bit signed integer raster maps (say, LCELL) could be added to GRASS, users would need to stick to GRASS since GDAL raster export of 64-bit integers is not available. Interesting idea."

A 64 bit signed integer type has been added to trunk with r72230. Various other projects, e.g. GDAL, SQLite, HDF5, Python also have some 64 bit signed integer type. In order to avoid namespace collision, the new type has been defined as grass_int64 and LCELL. GRASS libraries and modules do not yet make use of this type. Defining a 64 bit signed integer type has been tested on Linux + gcc, FreeBSD + clang, and MS Windows + gcc (mingw).

Great, thanks a lot, Markus !

What is the roadmap from here, not in terms of calendar, but in terms of tasks that need to be done to actually be able to use this new integer type ?

[rouault]

Related GDAL ticket: ​https://trac.osgeo.org/gdal/ticket/7052

[mmetz]

Replying to mlennert:

Replying to mmetz:

Replying to mlennert:

From #3084:

"Most importantly, a GRASS raster format for 64-bit signed integer. Note that GDAL does not support 64-bit signed or unsigned integers. The reason is probably that a portable implementation of 64-bit integers is not so easy. Regarding GRASS raster processing, the need for 64-bit integers usually arises only for raster maps with more than 2,147,483,647 cells which in turn usually requires Large File Support (LFS). Therefore the check for the availability of a 64-bit integer could be coupled to LFS. If support for 64-bit signed integer raster maps (say, LCELL) could be added to GRASS, users would need to stick to GRASS since GDAL raster export of 64-bit integers is not available. Interesting idea."

A 64 bit signed integer type has been added to trunk with r72230. Various other projects, e.g. GDAL, SQLite, HDF5, Python also have some 64 bit signed integer type. In order to avoid namespace collision, the new type has been defined as grass_int64 and LCELL. GRASS libraries and modules do not yet make use of this type. Defining a 64 bit signed integer type has been tested on Linux + gcc, FreeBSD + clang, and MS Windows + gcc (mingw).

Great, thanks a lot, Markus !

What is the roadmap from here, not in terms of calendar, but in terms of tasks that need to be done to actually be able to use this new integer type ?

Regarding internal variables, e.g. cell count, the first task would be to use the new 64 bit int type there instead of e.g. long which is sometimes 32 bit and sometimes 64 bit. I think there are still some places where int is used for cell counts. Making such variables 64 bit integer would be a bug fix, considering that large raster data are nowadays commonly processed.

Further on, some ifdefs like

#ifdef HAVE_LONG_LONG_INT
...
#endif

could be removed. Such ifdefs would need to be kept for printing messages, unless the length modifier j is available on all supported systems (no idea here).

A different issue is the introduction of a new LCELL raster map type. It should not be too difficult, but it would be a lot of work to add the corresponding functions, essentially cloning all the CELL type related functions and adjusting for LCELL.

Replying to rouault:

Related GDAL ticket: ​https://trac.osgeo.org/gdal/ticket/7052

Regarding nodata, GRASS does not have a global nodata type, but we would need new fns like

int Rast_is_l_null_value(const LCELL * lcellVal);
void Rast_set_l_null_value(LCELL * lcellVals, int numVals);

Still, such 64 bit integer GRASS raster maps could not be exported via GDAL.