Context Navigation

← Previous Ticket
Next Ticket →

#3927 closed defect (fixed)

r.external vs r.in.gdal under Windows

Reported by:	Nikos Alexandris	Owned by:	grass-dev@…
Priority:	critical	Milestone:	7.8.3
Component:	Raster	Version:	svn-releasebranch76
Keywords:	r.external, r.in.gdal	Cc:
CPU:	Unspecified	Platform:	MSWindows

Description

This is a case of a (hard to trace) buggy behavior when linking an external raster map of type Float32. Here an external report: https://gitlab.com/natcapes/r.estimap.recreation/issues/36.

In short, a raster map of type Float32, linked to an EPSG:3035 Location, viar.external, delivers different descriptive statistics under Windows than what the identical workflow does under Linux (and what gdalinfo tells about the data, both under Windows and Linux).

Also, the --overwrite flag given to either r.external or r.in.gdal does not seem to work under Windows.

Perhaps all these are known issues? Perhaps this case has to do with the presence of NULL cells too? As is the case of the example map I use.

The example map input_water_resources is available at https://gitlab.com/natcapes/r.estimap.recreation.data/blob/master/example_data_epsg_3035.tar.gz.

Change History (14)

comment:1 by Nikos Alexandris, 5 years ago

Maybe related somehow: https://trac.osgeo.org/grass/ticket/3857.

follow-ups: 4 8 comment:2 by hellik, 5 years ago

System Info                                                                     
GRASS Version: 7.6.1                                                            
GRASS SVN revision: r74292                                                      
Build date: 2019-03-19                                                          
Build platform: x86_64-w64-mingw32                                              
GDAL: 2.4.0                                                                     
PROJ.4: 5.2.0                                                                   
GEOS: 3.7.0                                                                     
SQLite: 3.26.0                                                                  
Python: 2.7.14                                                                  
wxPython: 2.8.12.1                                                              
Platform: Windows-10-10.0.18362 (OSGeo4W)

D:\temp\yoga\example_data_epsg_3035\raster>gdalinfo input_water_resources.tif -stats
Driver: GTiff/GeoTIFF
Files: input_water_resources.tif
Size is 2381, 2617
Coordinate System is:
PROJCS["ETRS89 / LAEA Europe",
    GEOGCS["ETRS89",
        DATUM["European_Terrestrial_Reference_System_1989",
            SPHEROID["GRS 1980",6378137,298.257222101,
                AUTHORITY["EPSG","7019"]],
            TOWGS84[0,0,0,0,0,0,0],
            AUTHORITY["EPSG","6258"]],
        PRIMEM["Greenwich",0,
            AUTHORITY["EPSG","8901"]],
        UNIT["degree",0.0174532925199433,
            AUTHORITY["EPSG","9122"]],
        AUTHORITY["EPSG","4258"]],
    PROJECTION["Lambert_Azimuthal_Equal_Area"],
    PARAMETER["latitude_of_center",52],
    PARAMETER["longitude_of_center",10],
    PARAMETER["false_easting",4321000],
    PARAMETER["false_northing",3210000],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    AUTHORITY["EPSG","3035"]]
Origin = (4735600.000000000000000,2879700.000000000000000)
Pixel Size = (50.000000000000000,-50.000000000000000)
Metadata:
  AREA_OR_POINT=Area
  TIFFTAG_SOFTWARE=GRASS GIS 7.9.dev with GDAL 2.3.1
Image Structure Metadata:
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  ( 4735600.000, 2879700.000) ( 15d39'28.78"E, 48d53'13.80"N)
Lower Left  ( 4735600.000, 2748850.000) ( 15d31'39.85"E, 47d42'49.53"N)
Upper Right ( 4854650.000, 2879700.000) ( 17d16'27.34"E, 48d47'35.80"N)
Lower Right ( 4854650.000, 2748850.000) ( 17d 6'26.37"E, 47d37'20.45"N)
Center      ( 4795125.000, 2814275.000) ( 16d23'29.04"E, 48d15'25.69"N)
Band 1 Block=2381x1 Type=Float32, ColorInterp=Gray
  Description = input_water_resources
  Minimum=0.000, Maximum=1.000, Mean=0.037, StdDev=0.103
  NoData Value=nan
  Metadata:
    COLOR_TABLE_RULES_COUNT=1
    COLOR_TABLE_RULE_RGB_0=0.000000e+00 1.000000e+00 255 255 255 0 0 255
    STATISTICS_MAXIMUM=1.0000001192093
    STATISTICS_MEAN=0.037087096658156
    STATISTICS_MINIMUM=0
    STATISTICS_STDDEV=0.10325697665504
    STATISTICS_VALID_PERCENT=57.84

r.in.gdal input=D:\temp\yoga\example_data_epsg_3035\raster\input_water_resources.tif output=input_water_resources_ringdal
Importing raster map <input_water_resources_ringdal>...

g.region -p -a raster=input_water_resources_ringdal@data align=input_water_resources_ringdal@data
projection: 99 (ETRS89 / LAEA Europe)
zone:       0
datum:      etrs89
ellipsoid:  grs80
north:      2879700
south:      2748850
west:       4735600
east:       4854650
nsres:      50
ewres:      50
rows:       2617
cols:       2381
cells:      6231077

r.univar -e map=input_water_resources_ringdal@data                              
total null and non-null cells: 6231077
total null cells: 2626801
Of the non-null cells:
----------------------
n: 3604276
minimum: 0
maximum: 1
range: 1
mean: 0.0370871
mean of absolute values: 0.0370871
standard deviation: 0.103257
variance: 0.010662
variation coefficient: 278.418 %
sum: 133672.132394677
1st quartile: 0
median (even number of cells): 0
3rd quartile: 0.000749337
90th percentile: 0.149497

r.external input=D:\temp\yoga\example_data_epsg_3035\raster\input_water_resources.tif output=input_water_resources_rexternal
Reading band 1 of 1...
Link to raster map <input_water_resources_rexternal> created.

r.univar -e map=input_water_resources_rexternal@data                            
total null and non-null cells: 6231077
total null cells: 5157779
Of the non-null cells:
----------------------
n: 1073298
minimum: 0.000101275
maximum: 1
range: 0.999899
mean: 0.124543
mean of absolute values: 0.124543
standard deviation: 0.157836
variance: 0.0249123
variation coefficient: 126.732 %
sum: 133672.132394677
1st quartile: 0.00224924
median (even number of cells): 0.0411837
3rd quartile: 0.234299
90th percentile: 0.329259

g.region -p -a raster=input_water_resources_rexternal@data align=input_water_resources_rexternal@data
projection: 99 (ETRS89 / LAEA Europe)
zone:       0
datum:      etrs89
ellipsoid:  grs80
north:      2879700
south:      2748850
west:       4735600
east:       4854650
nsres:      50
ewres:      50
rows:       2617
cols:       2381
cells:      6231077

r.univar -e map=input_water_resources_rexternal@data                            
total null and non-null cells: 6231077
total null cells: 5157779
Of the non-null cells:
----------------------
n: 1073298
minimum: 0.000101275
maximum: 1
range: 0.999899
mean: 0.124543
mean of absolute values: 0.124543
standard deviation: 0.157836
variance: 0.0249123
variation coefficient: 126.732 %
sum: 133672.132394677
1st quartile: 0.00224924
median (even number of cells): 0.0411837
3rd quartile: 0.234299
90th percentile: 0.329259

Perhaps all these are known issues? Perhaps this case has to do with the presence of NULL >cells too? As is the case of the example map I use.

r.univar -e map=input_water_resources_ringdal@data                              
total null and non-null cells: 6231077
total null cells: 2626801

r.univar -e map=input_water_resources_rexternal@data                            
total null and non-null cells: 6231077
total null cells: 5157779

null cells number seems to differ: r.in.gdal: 2626801 vs. r.external: total null cells: 5157779

comment:3 by hellik, 5 years ago

Priority:	major → critical

in reply to: 2 comment:4 by Nikos Alexandris, 5 years ago

Replying to hellik:

Perhaps all these are known issues? Perhaps this case has to do with the presence of NULL >cells too? As is the case of the example map I use.
r.univar -e map=input_water_resources_ringdal@data                              
total null and non-null cells: 6231077
total null cells: 2626801
vs
r.univar -e map=input_water_resources_rexternal@data                            
total null and non-null cells: 6231077
total null cells: 5157779
null cells number seems to differ: r.in.gdal: 2626801 vs. r.external: total null cells: 5157779

This confirms my findings. I should have went for the -e flag too, totally forgot it.

follow-up: 6 comment:5 by Nikos Alexandris, 5 years ago

In-Directly related-affected: https://github.com/qgis/QGIS/pull/32425

in reply to: 5 ; follow-up: 7 comment:6 by hellik, 5 years ago

Replying to Nikos Alexandris:

In-Directly related-affected: https://github.com/qgis/QGIS/pull/32425

why not fix it grass and let Qgis still use r.external?

in reply to: 6 comment:7 by Nikos Alexandris, 4 years ago

Replying to hellik:

Replying to Nikos Alexandris:

In-Directly related-affected: https://github.com/qgis/QGIS/pull/32425

why not fix it grass and let Qgis still use r.external?

Sure, when the fix is there. Professional deadlines, however, cannot wait for the real fix.

in reply to: 2 ; follow-up: 9 comment:8 by Nikos Alexandris, 4 years ago

Replying to hellik:

System Info                                                                     
GRASS Version: 7.6.1                                                            
GRASS SVN revision: r74292                                                      
Build date: 2019-03-19                                                          
Build platform: x86_64-w64-mingw32                                              
GDAL: 2.4.0                                                                     
PROJ.4: 5.2.0                                                                   
GEOS: 3.7.0                                                                     
SQLite: 3.26.0                                                                  
Python: 2.7.14                                                                  
wxPython: 2.8.12.1                                                              
Platform: Windows-10-10.0.18362 (OSGeo4W)

Helmut, this is the result of a command, or did you handcraft-ed these lines by copy-pasting?

in reply to: 8 ; follow-up: 10 comment:9 by hellik, 4 years ago

Replying to Nikos Alexandris:

Replying to hellik:

System Info                                                                     
GRASS Version: 7.6.1                                                            
GRASS SVN revision: r74292                                                      
Build date: 2019-03-19                                                          
Build platform: x86_64-w64-mingw32                                              
GDAL: 2.4.0                                                                     
PROJ.4: 5.2.0                                                                   
GEOS: 3.7.0                                                                     
SQLite: 3.26.0                                                                  
Python: 2.7.14                                                                  
wxPython: 2.8.12.1                                                              
Platform: Windows-10-10.0.18362 (OSGeo4W)

Helmut, this is the result of a command, or did you handcraft-ed these lines by copy-pasting?

Nothing hand-crafted, just go to the wxgui -> systeminfo

in reply to: 9 comment:10 by hellik, 4 years ago

Replying to hellik:

Replying to Nikos Alexandris:

Replying to hellik:

System Info                                                                     
GRASS Version: 7.6.1                                                            
GRASS SVN revision: r74292                                                      
Build date: 2019-03-19                                                          
Build platform: x86_64-w64-mingw32                                              
GDAL: 2.4.0                                                                     
PROJ.4: 5.2.0                                                                   
GEOS: 3.7.0                                                                     
SQLite: 3.26.0                                                                  
Python: 2.7.14                                                                  
wxPython: 2.8.12.1                                                              
Platform: Windows-10-10.0.18362 (OSGeo4W)

Helmut, this is the result of a command, or did you handcraft-ed these lines by copy-pasting?

Nothing hand-crafted, just go to the wxgui -> systeminfo

similar output by

g.version -r -e -g                                                              
version=7.8.1dev
date=2019
revision=4785a5b77
build_date=2019-10-25
build_platform=x86_64-w64-mingw32
build_off_t_size=8
libgis_revision=00000
libgis_date="?"
proj=5.2.0
gdal=2.4.1
geos=3.8.0
sqlite=3.29.0

comment:11 by mmetz, 4 years ago

Apparently 0 (zero) and NULL (nodata) got confused in r.external on Windows. If I set 0 (zero) to NULL (nodata), I get the exact same incorrect results. No idea where and why this confusion in r.external on Windows happens.

comment:12 by neteler, 4 years ago

Milestone:	→ 7.8.3

comment:13 by hellik, 4 years ago

fixed by


  Branch: refs/heads/master
  Home:   https://github.com/OSGeo/grass
  Commit: 87c52954e206ab06d2aaea52b0398d0dd4fea783
      https://github.com/OSGeo/grass/commit/87c52954e206ab06d2aaea52b0398d0dd4fea783
  Author: Markus Metz <33666869+metzm at users.noreply.github.com>
  Date:   2020-02-09 (Sun, 09 Feb 2020)

  Changed paths:
    M lib/raster/gdal.c

  Log Message:
  -----------
  r.external: read nan as nan (#338)

Fixes #329. atof() on windows can not read "nan" and returns 0 (zero) instead. Using Rast_set_d_null_value() in this case to set nan directly.

  Branch: refs/heads/releasebranch_7_8
  Home:   https://github.com/OSGeo/grass
  Commit: 607d3e126600c94ea31cbc8ff13974ad3e45a5b7
      https://github.com/OSGeo/grass/commit/607d3e126600c94ea31cbc8ff13974ad3e45a5b7
  Author: Markus Metz <33666869+metzm at users.noreply.github.com>
  Date:   2020-02-09 (Sun, 09 Feb 2020)

  Changed paths:
    M lib/raster/gdal.c

  Log Message:
  -----------
  r.external: read nan as nan (#338)

Fixes #329. atof() on windows can not read "nan" and returns 0 (zero) instead. Using Rast_set_d_null_value() in this case to set nan directly.

comment:14 by hellik, 4 years ago

Resolution:	→ fixed
Status:	new → closed

Note: See TracTickets for help on using tickets.

Download in other formats: