Ticket #1672 (closed defect: fixed)

Opened 1 year ago

Last modified 2 months ago

NaNs are not ignored when computing raster min/max

Reported by: jluis Assigned to: mloskot
Priority: normal Milestone: 1.6.0
Component: GDAL_Raster Version: 1.4.1
Severity: normal Keywords: nan infinite
Cc: warmerdam, rouault

Description

When raster floats have NaNs?, they are not ignored by the min/max computation code. This causes stats reports like this Minimum=-1.#IO, Maximum=-1.#IO, Mean=-1.#IO, StdDev?=-1.#IO

Metadata:

STATISTICS_MINIMUM=-1.#IND STATISTICS_MAXIMUM=-1.#IND STATISTICS_MEAN=-1.#IND STATISTICS_STDDEV=-1.#IND

Attachments

gdal_svn_trunk_nans_ignore_1672.patch (0.8 kB) - added by rouault on 11/20/07 18:13:18.
stats_and_timings_before_patch.txt (3.7 kB) - added by rouault on 11/20/07 18:14:56.
stats_and_timings_after_patch.txt (4.1 kB) - added by rouault on 11/20/07 18:15:10.
generate_nans.cpp (2.2 kB) - added by rouault on 11/20/07 18:15:41.
Code to generate files with Nans
gdal_misc.cpp-mloskot.patch (1.3 kB) - added by mloskot on 12/18/07 22:28:08.
In my opinion, this function should be patched too

Change History

06/14/07 11:00:56 changed by warmerdam

  • keywords set to nan.
  • owner changed from warmerdam to mloskot.
  • cc set to warmerdam.
  • milestone set to 1.5.0.

Mateusz,

I would like you to try and address this for 1.5.0. We should prepare some test files with NaNs?, and ensure they will be ignored by the generic min/max computation code, and the stats code.

Please do some review of the performance effects of this change (is the NaN test expensive?) and ensure this feature gets tested in the test suite.

It should be possible to use the CPLIsNan() macro to test for NaN.

This checking should of course only be done for float and double pixel types.

11/20/07 18:12:52 changed by rouault

I'm attaching a patch (quite trivial) do do the CPLIsNan test in GDALRasterBand::ComputeStatistics?. (I didn't find a "generic min/max computation code")

I've tested it on a 10000x10000 float32 GTiff and 10000*5000 float64 GTiff. The timings are a bit difficult to interpret as there is a lot of I/O implied. I think we must consider the 'user' part of the timing. They would show that in the 32bit case, CPU time is about 2x longer. But for 64bit case, no significant difference. I'm a bit skeptical on my timing method...

11/20/07 18:13:18 changed by rouault

  • attachment gdal_svn_trunk_nans_ignore_1672.patch added.

11/20/07 18:14:56 changed by rouault

  • attachment stats_and_timings_before_patch.txt added.

11/20/07 18:15:10 changed by rouault

  • attachment stats_and_timings_after_patch.txt added.

11/20/07 18:15:41 changed by rouault

  • attachment generate_nans.cpp added.

Code to generate files with Nans

11/24/07 05:28:00 changed by rouault

  • summary changed from NaNs are not ignored when computing raster min/max to [PATCH] NaNs are not ignored when computing raster min/max.

12/18/07 22:28:08 changed by mloskot

  • attachment gdal_misc.cpp-mloskot.patch added.

In my opinion, this function should be patched too

12/18/07 23:14:30 changed by mloskot

  • keywords changed from nan to nan infinite.
  • status changed from new to assigned.

12/18/07 23:14:35 changed by mloskot

  • status changed from assigned to closed.
  • resolution set to fixed.

Fixed in trunk (r13382) and branches/1.5 (r13384). Thanks to Even for the patch!

02/08/08 08:38:39 changed by warmerdam

  • status changed from closed to reopened.
  • cc changed from warmerdam to warmerdam, rouault.
  • summary changed from [PATCH] NaNs are not ignored when computing raster min/max to NaNs are not ignored when computing raster min/max.
  • resolution deleted.
  • milestone deleted.

We are getting the following errors in the VC7.1 build on Tamas' buildbot. We need to fix one way or another. 

 ------------ Failures ------------
Script: gcore/stats.py
  TEST: stats_nan_1 ... fail
    Checksum for band 1 in "nan32.tif" is 978, but expected 874.
  TEST: stats_nan_2 ... fail
    Checksum for band 1 in "nan64.tif" is 4942, but expected 4414.
 ----------------------------------

02/28/08 16:21:02 changed by rouault

  • status changed from reopened to closed.
  • resolution set to fixed.
  • milestone set to 1.6.0.

Fixed in trunk in r13893