Opened 7 years ago

Closed 7 years ago

Last modified 6 years ago

#5868 closed enhancement (fixed)

[PATCH] Add Quartiles, Min & Max to gdalwarp

Reported by: davisnn Owned by: warmerdam
Priority: normal Milestone: 2.0.0
Component: GDAL_Raster Version: 1.11.1
Severity: normal Keywords:
Cc:

Description

In addition to the Average and Mode statistics it is helpful for several of my projects to be able to extract the quartiles, max and min when warping the data.

I have attached a patch to gdal-1.11.1 that adds this functionality to gdalwarp.

Attachments (6)

quartiles.patch (11.8 KB) - added by davisnn 7 years ago.
Patch to add quartiles, min, and max to gdalwarp
quartiles_r1.patch (16.3 KB) - added by davisnn 7 years ago.
Patch with changes to address deficiencies
quartiles_r2.patch (17.4 KB) - added by davisnn 7 years ago.
Patch fixing DBL_MIN and printf errors
quantile_data.tar.gz (11.5 KB) - added by davisnn 7 years ago.
Data for quantile autotest
autotest.patch (5.4 KB) - added by davisnn 7 years ago.
Patch to add tests for max, min, median, Quartile 1 and Quartile3 to autotest
quantile_data_r1.tar.gz (11.5 KB) - added by davisnn 7 years ago.
Data for autotest with updated VRT files for negative value tests

Download all attachments as: .zip

Change History (15)

Changed 7 years ago by davisnn

Attachment: quartiles.patch added

Patch to add quartiles, min, and max to gdalwarp

comment:1 Changed 7 years ago by Even Rouault

Summary: Add Quartiles, Min & Max to gdalwarp[PATCH] Add Quartiles, Min & Max to gdalwarp

Several remarks :

  • this should be rebased on GDAL trunk. Note that the value 7 of GDALResampleAlg is now reserved for another purpose
  • the numeric values algo = 4, 5, 6 are not very intuitive. I know this has started like this, but adding more to it increases lack of readability. Symbolic names would be better
  • initialization to dfTotal = 1e6; for the min algorithm is arbitrary. Better use DBL_MAX (or std::limits<double>::max or whatever works)
  • same for initialization to dfTotal = 0. Use -DBL_MAX
  • nCount2++; is useless in all places where it is added
  • gdalwarp Usage() should be updated with -rmx, etc... but I found the new dedicated options a bit too specialized, so perhaps just remove them and keep the -r max syntax.
  • apps/gdal_utilities.dox should be updated
  • autotest/alg/warp.py should be updated to test the new algos

comment:2 Changed 7 years ago by davisnn

Thank you for the feedback. I have made the suggested changes, except for the changes to autotest (quartiles_r1.patch). I was not able to find much in the way of documentation of how to add a test to the autotest. Do I just need to duplicate what is being done for the mode cases in say warp_36?

Also I was not quite sure what you meant by symbolic names. I have added an enum to gdalwarper.h, that sets names to each of the algorithms, if this is not what you meant please let me know.

Thanks, Neil

Changed 7 years ago by davisnn

Attachment: quartiles_r1.patch added

Patch with changes to address deficiencies

comment:3 Changed 7 years ago by Even Rouault

  • DBL_MIN is not appropriate. It is the smallest positive number something like 1e-XXXX. That should rather be -DBL_MAX, otherwise that will not work with negative only values
  • There are remaining printf() statements.
  • Yes, you can take example on the mode case.

Changed 7 years ago by davisnn

Attachment: quartiles_r2.patch added

Patch fixing DBL_MIN and printf errors

Changed 7 years ago by davisnn

Attachment: quantile_data.tar.gz added

Data for quantile autotest

Changed 7 years ago by davisnn

Attachment: autotest.patch added

Patch to add tests for max, min, median, Quartile 1 and Quartile3 to autotest

comment:4 in reply to:  3 Changed 7 years ago by davisnn

I have removed the two errors you mentioned and added test cases for both signed and unsigned files to the autotest.

comment:5 Changed 7 years ago by Even Rouault

Hum, I get failed tests with the new test steps (on a x86_64 with 4.8.2 in both -00 and -02 mode), and the visual difference is indeed huge between what I get from the vrt with gdal_translate and the reference tif, not just a subtle issue with floating point rounding. Are you sure the reference tif are uptodate in your archive ?

  TEST: warp_47 ... Diff at pixel (1, 0) : -9.000000
Diff at pixel (2, 0) : -33.000000
Diff at pixel (3, 0) : -50.000000
Diff at pixel (4, 0) : -33.000000
Diff at pixel (5, 0) : -9.000000
Diff at pixel (12, 0) : -24.000000
Diff at pixel (15, 0) : -16.000000
Diff at pixel (16, 0) : -8.000000
Diff at pixel (17, 0) : -16.000000
Diff at pixel (6, 2) : -66.000000
Diff at pixel (9, 2) : -99.000000
Diff at pixel (29, 7) : -123.000000
Diff at pixel (2, 12) : -132.000000
Max diff : 132
Number of diffs : 589
fail
    line 1540: Image too different from reference
  TEST: warp_48 ... Diff at pixel (0, 0) : 25.000000
Diff at pixel (1, 0) : 16.000000
Diff at pixel (5, 0) : 16.000000
Diff at pixel (6, 0) : 33.000000
Diff at pixel (7, 0) : 25.000000
Diff at pixel (8, 0) : 17.000000
Diff at pixel (9, 0) : 57.000000
Diff at pixel (10, 0) : 33.000000
Diff at pixel (11, 0) : 8.000000
Diff at pixel (13, 0) : 58.000000
Diff at pixel (8, 3) : 82.000000
Diff at pixel (28, 4) : 99.000000
Diff at pixel (4, 11) : 107.000000
Diff at pixel (2, 13) : 132.000000
Diff at pixel (2, 16) : 140.000000
Diff at pixel (27, 25) : 148.000000
Max diff : 148
Number of diffs : 615
fail
    line 1559: Image too different from reference
  TEST: warp_49 ... Diff at pixel (0, 0) : 17.000000
Diff at pixel (1, 0) : 8.000000
Diff at pixel (2, 0) : -17.000000
Diff at pixel (3, 0) : -17.000000
Diff at pixel (4, 0) : -25.000000
Diff at pixel (6, 0) : 25.000000
Diff at pixel (7, 0) : 9.000000
Diff at pixel (8, 0) : 9.000000
Diff at pixel (9, 0) : 8.000000
Diff at pixel (13, 0) : 33.000000
Diff at pixel (13, 1) : 49.000000
Diff at pixel (26, 4) : 57.000000
Diff at pixel (27, 4) : 74.000000
Diff at pixel (4, 11) : 107.000000
Diff at pixel (2, 16) : 140.000000
Max diff : 140
Number of diffs : 579
fail
    line 1578: Image too different from reference
  TEST: warp_50 ... Diff at pixel (0, 0) : 25.000000
Diff at pixel (1, 0) : 16.000000
Diff at pixel (5, 0) : 16.000000
Diff at pixel (6, 0) : 33.000000
Diff at pixel (7, 0) : 25.000000
Diff at pixel (8, 0) : 17.000000
Diff at pixel (9, 0) : 57.000000
Diff at pixel (10, 0) : 33.000000
Diff at pixel (11, 0) : 8.000000
Diff at pixel (13, 0) : 58.000000
Diff at pixel (8, 3) : 82.000000
Diff at pixel (28, 4) : 99.000000
Diff at pixel (4, 11) : 107.000000
Diff at pixel (2, 13) : 132.000000
Diff at pixel (2, 16) : 140.000000
Diff at pixel (27, 25) : 148.000000
Max diff : 148
Number of diffs : 615
fail
    line 1597: Image too different from reference
  TEST: warp_51 ... Diff at pixel (0, 0) : 9.000000
Diff at pixel (2, 0) : -17.000000
Diff at pixel (3, 0) : -25.000000
Diff at pixel (4, 0) : -25.000000
Diff at pixel (10, 0) : 8.000000
Diff at pixel (12, 0) : -16.000000
Diff at pixel (13, 0) : 25.000000
Diff at pixel (15, 0) : -16.000000
Diff at pixel (17, 0) : -8.000000
Diff at pixel (24, 0) : -33.000000
Diff at pixel (13, 1) : 41.000000
Diff at pixel (6, 2) : -57.000000
Diff at pixel (28, 4) : 58.000000
Diff at pixel (26, 6) : -99.000000
Diff at pixel (29, 7) : -123.000000
Diff at pixel (2, 12) : -132.000000
Max diff : 132
Number of diffs : 542
fail
    line 1616: Image too different from reference

comment:6 Changed 7 years ago by davisnn

The problem was actually with the VRT files. I have uploaded a new copy of the data, with the updated VRT files. This passes all of the tests.

Changed 7 years ago by davisnn

Attachment: quantile_data_r1.tar.gz added

Data for autotest with updated VRT files for negative value tests

comment:7 Changed 7 years ago by Even Rouault

trunk r28717 "warper/gdalwarp: add min,max,med,q1 and q3 resampling algorithms (patch by davisnn, #5868)"

comment:8 Changed 7 years ago by Even Rouault

Milestone: 2.0
Resolution: fixed
Status: newclosed

comment:9 Changed 6 years ago by Even Rouault

Milestone: 2.02.0.0

Milestone renamed

Note: See TracTickets for help on using tickets.