Ticket #1672 (closed defect: fixed)

Opened 5 years ago

Last modified 4 years ago

NaNs are not ignored when computing raster min/max

Reported by: jluis Owned by: mloskot
Priority: normal Milestone: 1.6.0
Component: GDAL_Raster Version: 1.4.1
Severity: normal Keywords: nan infinite
Cc: warmerdam, rouault

Description

When raster floats have NaNs?, they are not ignored by the min/max computation code. This causes stats reports like this Minimum=-1.#IO, Maximum=-1.#IO, Mean=-1.#IO, StdDev?=-1.#IO

Metadata:

STATISTICS_MINIMUM=-1.#IND STATISTICS_MAXIMUM=-1.#IND STATISTICS_MEAN=-1.#IND STATISTICS_STDDEV=-1.#IND

Attachments

gdal_svn_trunk_nans_ignore_1672.patch Download (0.8 KB) - added by rouault 4 years ago.
stats_and_timings_before_patch.txt Download (3.7 KB) - added by rouault 4 years ago.
stats_and_timings_after_patch.txt Download (4.1 KB) - added by rouault 4 years ago.
generate_nans.cpp Download (2.2 KB) - added by rouault 4 years ago.
Code to generate files with Nans
gdal_misc.cpp-mloskot.patch Download (1.3 KB) - added by mloskot 4 years ago.
In my opinion, this function should be patched too

Change History

Changed 5 years ago by warmerdam

  • cc warmerdam added
  • keywords nan added
  • owner changed from warmerdam to mloskot
  • milestone set to 1.5.0

Mateusz,

I would like you to try and address this for 1.5.0. We should prepare some test files with NaNs?, and ensure they will be ignored by the generic min/max computation code, and the stats code.

Please do some review of the performance effects of this change (is the NaN test expensive?) and ensure this feature gets tested in the test suite.

It should be possible to use the CPLIsNan() macro to test for NaN.

This checking should of course only be done for float and double pixel types.

Changed 4 years ago by rouault

I'm attaching a patch (quite trivial) do do the CPLIsNan test in GDALRasterBand::ComputeStatistics?. (I didn't find a "generic min/max computation code")

I've tested it on a 10000x10000 float32 GTiff and 10000*5000 float64 GTiff. The timings are a bit difficult to interpret as there is a lot of I/O implied. I think we must consider the 'user' part of the timing. They would show that in the 32bit case, CPU time is about 2x longer. But for 64bit case, no significant difference. I'm a bit skeptical on my timing method...

Changed 4 years ago by rouault

Changed 4 years ago by rouault

Changed 4 years ago by rouault

Changed 4 years ago by rouault

Code to generate files with Nans

Changed 4 years ago by rouault

  • summary changed from NaNs are not ignored when computing raster min/max to [PATCH] NaNs are not ignored when computing raster min/max

Changed 4 years ago by mloskot

In my opinion, this function should be patched too

Changed 4 years ago by mloskot

  • keywords infinite added
  • status changed from new to assigned

Changed 4 years ago by mloskot

  • status changed from assigned to closed
  • resolution set to fixed

Fixed in trunk (r13382) and branches/1.5 (r13384). Thanks to Even for the patch!

Changed 4 years ago by warmerdam

  • cc rouault added
  • status changed from closed to reopened
  • summary changed from [PATCH] NaNs are not ignored when computing raster min/max to NaNs are not ignored when computing raster min/max
  • resolution fixed deleted
  • milestone 1.5.0 deleted

We are getting the following errors in the VC7.1 build on Tamas' buildbot. We need to fix one way or another. 

 ------------ Failures ------------
Script: gcore/stats.py
  TEST: stats_nan_1 ... fail
    Checksum for band 1 in "nan32.tif" is 978, but expected 874.
  TEST: stats_nan_2 ... fail
    Checksum for band 1 in "nan64.tif" is 4942, but expected 4414.
 ----------------------------------

Changed 4 years ago by rouault

  • status changed from reopened to closed
  • resolution set to fixed
  • milestone set to 1.6.0

Fixed in trunk in r13893

Note: See TracTickets for help on using tickets.