Opened 12 years ago

Closed 11 years ago

#1672 closed defect (fixed)

NaNs are not ignored when computing raster min/max

Reported by: jluis Owned by: Mateusz Łoskot
Priority: normal Milestone: 1.6.0
Component: GDAL_Raster Version: 1.4.1
Severity: normal Keywords: nan infinite
Cc: warmerdam, Even Rouault

Description

When raster floats have NaNs?, they are not ignored by the min/max computation code. This causes stats reports like this Minimum=-1.#IO, Maximum=-1.#IO, Mean=-1.#IO, StdDev?=-1.#IO

Metadata:

STATISTICS_MINIMUM=-1.#IND STATISTICS_MAXIMUM=-1.#IND STATISTICS_MEAN=-1.#IND STATISTICS_STDDEV=-1.#IND

Attachments (5)

gdal_svn_trunk_nans_ignore_1672.patch (799 bytes) - added by Even Rouault 11 years ago.
stats_and_timings_before_patch.txt (3.7 KB) - added by Even Rouault 11 years ago.
stats_and_timings_after_patch.txt (4.1 KB) - added by Even Rouault 11 years ago.
generate_nans.cpp (2.2 KB) - added by Even Rouault 11 years ago.
Code to generate files with Nans
gdal_misc.cpp-mloskot.patch (1.3 KB) - added by Mateusz Łoskot 11 years ago.
In my opinion, this function should be patched too

Download all attachments as: .zip

Change History (12)

comment:1 Changed 12 years ago by warmerdam

Cc: warmerdam added
Keywords: nan added
Milestone: 1.5.0
Owner: changed from warmerdam to Mateusz Łoskot

Mateusz,

I would like you to try and address this for 1.5.0. We should prepare some test files with NaNs?, and ensure they will be ignored by the generic min/max computation code, and the stats code.

Please do some review of the performance effects of this change (is the NaN test expensive?) and ensure this feature gets tested in the test suite.

It should be possible to use the CPLIsNan() macro to test for NaN.

This checking should of course only be done for float and double pixel types.

comment:2 Changed 11 years ago by Even Rouault

I'm attaching a patch (quite trivial) do do the CPLIsNan test in GDALRasterBand::ComputeStatistics?. (I didn't find a "generic min/max computation code")

I've tested it on a 10000x10000 float32 GTiff and 10000*5000 float64 GTiff. The timings are a bit difficult to interpret as there is a lot of I/O implied. I think we must consider the 'user' part of the timing. They would show that in the 32bit case, CPU time is about 2x longer. But for 64bit case, no significant difference. I'm a bit skeptical on my timing method...

Changed 11 years ago by Even Rouault

Changed 11 years ago by Even Rouault

Changed 11 years ago by Even Rouault

Changed 11 years ago by Even Rouault

Attachment: generate_nans.cpp added

Code to generate files with Nans

comment:3 Changed 11 years ago by Even Rouault

Summary: NaNs are not ignored when computing raster min/max[PATCH] NaNs are not ignored when computing raster min/max

Changed 11 years ago by Mateusz Łoskot

Attachment: gdal_misc.cpp-mloskot.patch added

In my opinion, this function should be patched too

comment:4 Changed 11 years ago by Mateusz Łoskot

Keywords: infinite added
Status: newassigned

comment:5 Changed 11 years ago by Mateusz Łoskot

Resolution: fixed
Status: assignedclosed

Fixed in trunk (r13382) and branches/1.5 (r13384). Thanks to Even for the patch!

comment:6 Changed 11 years ago by warmerdam

Cc: Even Rouault added
Milestone: 1.5.0
Resolution: fixed
Status: closedreopened
Summary: [PATCH] NaNs are not ignored when computing raster min/maxNaNs are not ignored when computing raster min/max

We are getting the following errors in the VC7.1 build on Tamas' buildbot. We need to fix one way or another.

 ------------ Failures ------------
Script: gcore/stats.py
  TEST: stats_nan_1 ... fail
    Checksum for band 1 in "nan32.tif" is 978, but expected 874.
  TEST: stats_nan_2 ... fail
    Checksum for band 1 in "nan64.tif" is 4942, but expected 4414.
 ----------------------------------

comment:7 Changed 11 years ago by Even Rouault

Milestone: 1.6.0
Resolution: fixed
Status: reopenedclosed

Fixed in trunk in r13893

Note: See TracTickets for help on using tickets.