[PATCH] Add Quartiles, Min & Max to gdalwarp

[davisnn]

In addition to the Average and Mode statistics it is helpful for several of my projects to be able to extract the quartiles, max and min when warping the data.

I have attached a patch to gdal-1.11.1 that adds this functionality to gdalwarp.

[davisnn]

Patch to add quartiles, min, and max to gdalwarp

[Even Rouault]

Several remarks :

• this should be rebased on GDAL trunk. Note that the value 7 of GDALResampleAlg is now reserved for another purpose
• the numeric values algo = 4, 5, 6 are not very intuitive. I know this has started like this, but adding more to it increases lack of readability. Symbolic names would be better
• initialization to dfTotal = 1e6; for the min algorithm is arbitrary. Better use DBL_MAX (or std::limits<double>::max or whatever works)
• same for initialization to dfTotal = 0. Use -DBL_MAX
• nCount2++; is useless in all places where it is added
• gdalwarp Usage() should be updated with -rmx, etc... but I found the new dedicated options a bit too specialized, so perhaps just remove them and keep the -r max syntax.
• apps/gdal_utilities.dox should be updated
• autotest/alg/warp.py should be updated to test the new algos

[davisnn]

Thank you for the feedback. I have made the suggested changes, except for the changes to autotest (quartiles_r1.patch). I was not able to find much in the way of documentation of how to add a test to the autotest. Do I just need to duplicate what is being done for the mode cases in say warp_36?

Also I was not quite sure what you meant by symbolic names. I have added an enum to gdalwarper.h, that sets names to each of the algorithms, if this is not what you meant please let me know.

Thanks, Neil

[davisnn]

Patch with changes to address deficiencies

[Even Rouault]

• DBL_MIN is not appropriate. It is the smallest positive number something like 1e-XXXX. That should rather be -DBL_MAX, otherwise that will not work with negative only values
• There are remaining printf() statements.
• Yes, you can take example on the mode case.

[davisnn]

Patch fixing DBL_MIN and printf errors

[davisnn]

Data for quantile autotest

[davisnn]

Patch to add tests for max, min, median, Quartile 1 and Quartile3 to autotest

[davisnn]

I have removed the two errors you mentioned and added test cases for both signed and unsigned files to the autotest.

[Even Rouault]

Hum, I get failed tests with the new test steps (on a x86_64 with 4.8.2 in both -00 and -02 mode), and the visual difference is indeed huge between what I get from the vrt with gdal_translate and the reference tif, not just a subtle issue with floating point rounding. Are you sure the reference tif are uptodate in your archive ?

  TEST: warp_47 ... Diff at pixel (1, 0) : -9.000000
Diff at pixel (2, 0) : -33.000000
Diff at pixel (3, 0) : -50.000000
Diff at pixel (4, 0) : -33.000000
Diff at pixel (5, 0) : -9.000000
Diff at pixel (12, 0) : -24.000000
Diff at pixel (15, 0) : -16.000000
Diff at pixel (16, 0) : -8.000000
Diff at pixel (17, 0) : -16.000000
Diff at pixel (6, 2) : -66.000000
Diff at pixel (9, 2) : -99.000000
Diff at pixel (29, 7) : -123.000000
Diff at pixel (2, 12) : -132.000000
Max diff : 132
Number of diffs : 589
fail
    line 1540: Image too different from reference
  TEST: warp_48 ... Diff at pixel (0, 0) : 25.000000
Diff at pixel (1, 0) : 16.000000
Diff at pixel (5, 0) : 16.000000
Diff at pixel (6, 0) : 33.000000
Diff at pixel (7, 0) : 25.000000
Diff at pixel (8, 0) : 17.000000
Diff at pixel (9, 0) : 57.000000
Diff at pixel (10, 0) : 33.000000
Diff at pixel (11, 0) : 8.000000
Diff at pixel (13, 0) : 58.000000
Diff at pixel (8, 3) : 82.000000
Diff at pixel (28, 4) : 99.000000
Diff at pixel (4, 11) : 107.000000
Diff at pixel (2, 13) : 132.000000
Diff at pixel (2, 16) : 140.000000
Diff at pixel (27, 25) : 148.000000
Max diff : 148
Number of diffs : 615
fail
    line 1559: Image too different from reference
  TEST: warp_49 ... Diff at pixel (0, 0) : 17.000000
Diff at pixel (1, 0) : 8.000000
Diff at pixel (2, 0) : -17.000000
Diff at pixel (3, 0) : -17.000000
Diff at pixel (4, 0) : -25.000000
Diff at pixel (6, 0) : 25.000000
Diff at pixel (7, 0) : 9.000000
Diff at pixel (8, 0) : 9.000000
Diff at pixel (9, 0) : 8.000000
Diff at pixel (13, 0) : 33.000000
Diff at pixel (13, 1) : 49.000000
Diff at pixel (26, 4) : 57.000000
Diff at pixel (27, 4) : 74.000000
Diff at pixel (4, 11) : 107.000000
Diff at pixel (2, 16) : 140.000000
Max diff : 140
Number of diffs : 579
fail
    line 1578: Image too different from reference
  TEST: warp_50 ... Diff at pixel (0, 0) : 25.000000
Diff at pixel (1, 0) : 16.000000
Diff at pixel (5, 0) : 16.000000
Diff at pixel (6, 0) : 33.000000
Diff at pixel (7, 0) : 25.000000
Diff at pixel (8, 0) : 17.000000
Diff at pixel (9, 0) : 57.000000
Diff at pixel (10, 0) : 33.000000
Diff at pixel (11, 0) : 8.000000
Diff at pixel (13, 0) : 58.000000
Diff at pixel (8, 3) : 82.000000
Diff at pixel (28, 4) : 99.000000
Diff at pixel (4, 11) : 107.000000
Diff at pixel (2, 13) : 132.000000
Diff at pixel (2, 16) : 140.000000
Diff at pixel (27, 25) : 148.000000
Max diff : 148
Number of diffs : 615
fail
    line 1597: Image too different from reference
  TEST: warp_51 ... Diff at pixel (0, 0) : 9.000000
Diff at pixel (2, 0) : -17.000000
Diff at pixel (3, 0) : -25.000000
Diff at pixel (4, 0) : -25.000000
Diff at pixel (10, 0) : 8.000000
Diff at pixel (12, 0) : -16.000000
Diff at pixel (13, 0) : 25.000000
Diff at pixel (15, 0) : -16.000000
Diff at pixel (17, 0) : -8.000000
Diff at pixel (24, 0) : -33.000000
Diff at pixel (13, 1) : 41.000000
Diff at pixel (6, 2) : -57.000000
Diff at pixel (28, 4) : 58.000000
Diff at pixel (26, 6) : -99.000000
Diff at pixel (29, 7) : -123.000000
Diff at pixel (2, 12) : -132.000000
Max diff : 132
Number of diffs : 542
fail
    line 1616: Image too different from reference

[davisnn]

The problem was actually with the VRT files. I have uploaded a new copy of the data, with the updated VRT files. This passes all of the tests.

[davisnn]

Data for autotest with updated VRT files for negative value tests

[Even Rouault]

trunk r28717 "warper/gdalwarp: add min,max,med,q1 and q3 resampling algorithms (patch by davisnn, #5868)"

[Even Rouault]

Milestone renamed