Opened 17 years ago

Closed 16 years ago

#1672 closed defect (fixed)

NaNs are not ignored when computing raster min/max

Reported by: jluis Owned by: Mateusz Łoskot
Priority: normal Milestone: 1.6.0
Component: GDAL_Raster Version: 1.4.1
Severity: normal Keywords: nan infinite
Cc: warmerdam, Even Rouault

Description

When raster floats have NaNs, they are not ignored by the min/max computation code. This causes stats reports like this Minimum=-1.#IO, Maximum=-1.#IO, Mean=-1.#IO, StdDev=-1.#IO

Metadata:

STATISTICS_MINIMUM=-1.#IND STATISTICS_MAXIMUM=-1.#IND STATISTICS_MEAN=-1.#IND STATISTICS_STDDEV=-1.#IND

Attachments (5)

gdal_svn_trunk_nans_ignore_1672.patch​ (799 bytes ) - added by Even Rouault 16 years ago.
stats_and_timings_before_patch.txt​ (3.7 KB ) - added by Even Rouault 16 years ago.
stats_and_timings_after_patch.txt​ (4.1 KB ) - added by Even Rouault 16 years ago.
generate_nans.cpp​ (2.2 KB ) - added by Even Rouault 16 years ago.
Code to generate files with Nans
gdal_misc.cpp-mloskot.patch​ (1.3 KB ) - added by Mateusz Łoskot 16 years ago.
In my opinion, this function should be patched too

Download all attachments as: .zip

Change History (12)

comment:1 by warmerdam, 17 years ago

Cc: warmerdam added
Keywords: nan added
Milestone: → 1.5.0
Owner: changed from warmerdam to Mateusz Łoskot

Mateusz,

I would like you to try and address this for 1.5.0. We should prepare some test files with NaNs, and ensure they will be ignored by the generic min/max computation code, and the stats code.

Please do some review of the performance effects of this change (is the NaN test expensive?) and ensure this feature gets tested in the test suite.

It should be possible to use the CPLIsNan() macro to test for NaN.

This checking should of course only be done for float and double pixel types.

comment:2 by Even Rouault, 16 years ago

I'm attaching a patch (quite trivial) do do the CPLIsNan test in GDALRasterBand::ComputeStatistics. (I didn't find a "generic min/max computation code")

I've tested it on a 10000x10000 float32 GTiff and 10000*5000 float64 GTiff. The timings are a bit difficult to interpret as there is a lot of I/O implied. I think we must consider the 'user' part of the timing. They would show that in the 32bit case, CPU time is about 2x longer. But for 64bit case, no significant difference. I'm a bit skeptical on my timing method...

by Even Rouault, 16 years ago

Attachment: gdal_svn_trunk_nans_ignore_1672.patch​ added

by Even Rouault, 16 years ago

Attachment: stats_and_timings_before_patch.txt​ added

by Even Rouault, 16 years ago

Attachment: stats_and_timings_after_patch.txt​ added

by Even Rouault, 16 years ago

Attachment: generate_nans.cpp​ added

Code to generate files with Nans

comment:3 by Even Rouault, 16 years ago

Summary: NaNs are not ignored when computing raster min/max → [PATCH] NaNs are not ignored when computing raster min/max

by Mateusz Łoskot, 16 years ago

Attachment: gdal_misc.cpp-mloskot.patch​ added

In my opinion, this function should be patched too

comment:4 by Mateusz Łoskot, 16 years ago

Keywords: infinite added
Status: new → assigned

comment:5 by Mateusz Łoskot, 16 years ago

Resolution: → fixed
Status: assigned → closed

Fixed in trunk (r13382) and branches/1.5 (r13384). Thanks to Even for the patch!

comment:6 by warmerdam, 16 years ago

Cc: Even Rouault added
Milestone: 1.5.0
Resolution: fixed
Status: closed → reopened
Summary: [PATCH] NaNs are not ignored when computing raster min/max → NaNs are not ignored when computing raster min/max

We are getting the following errors in the VC7.1 build on Tamas' buildbot. We need to fix one way or another. 

 ------------ Failures ------------
Script: gcore/stats.py
  TEST: stats_nan_1 ... fail
    Checksum for band 1 in "nan32.tif" is 978, but expected 874.
  TEST: stats_nan_2 ... fail
    Checksum for band 1 in "nan64.tif" is 4942, but expected 4414.
 ----------------------------------

comment:7 by Even Rouault, 16 years ago

Milestone: → 1.6.0
Resolution: → fixed
Status: reopened → closed

Fixed in trunk in r13893

Note: See TracTickets for help on using tickets.