[PATCH] gdalwarp speed-up when warping small images into a big mosaic

[peifer]

Here a summary of some discussion at gdal-dev@…

Adam Nowacki wrote:
Some rather counterintuitive gdalwarp behavior: the bigger dfWarpMemoryLimit (-wm setting) the more cpu time will be wasted on warping not existing pixels. Why? Warping begins with the destination window size of entire output image size. If this size is larger than dfWarpMemoryLimit it is split in half along the longest axis and any half that doesn't contain the currently processed source file is discarded. With large dfWarpMemoryLimit this subdivision process will stop early with still large portions of out of source image pixels.

My observation while gdalwarping *many* small GeoTiffs into a big mosaic, with -wm 300:

Src=0,0,3601x3601 Dst=36375,31125,12125x10375

I understand this as follows: the source image of 3601x3601 pixels is read as one chunk (which after reprojection and resampling should be around 800x1100 pixels). However, 12125x10375 pixels are actually written to the output file. So 99.5% of the destination window must be completely unrelated to my input image.

If I counter-intuitively reduce the memory limit to, say: 40 MB and leave anything else unchanged, then gdalwarp is processing the input files much faster.

Perhaps gdalwarp should not only test if the destination window fits into memory, but also check what would be the minimum destination window for warping the input image. This could speed up the mosaicing of small tiles into a bigger output file.

Adam provided a patch with a new variable dfFillRatio, which made things even faster. The patch is here: ​http://pastebin.com/m45e46f53

[Even Rouault]

Patch from ​http://pastebin.com/m45e46f53

[peifer]

Even,

(Frank speaking here with Hermann)

If you are confident about this change, just go ahead and apply it.

[Jukka Rahkonen]

Patch not applied yet. Recent mailing list thread shows that problem with counterintuitive behaviour still exist ​http://thread.gmane.org/gmane.comp.gis.gdal.devel/39250.

[peifer]

IIRC: At the time (in other words: 5 years ago) I applied the patch locally and it was working fine. I never understood why it has been ignored.

[Even Rouault]

Replying to peifer:

IIRC: At the time (in other words: 5 years ago) I applied the patch locally and it was working fine. I never understood why it has been ignored.

Well, I spent more than 2 hours of unfunded time reviewing, testing, fixing/improving the initial patch and adding a regression test for that issue... If there are issues you'd wish to be fixed more quickly, there are developers who can be sponsored for that.

trunk r28258 "warper: regardless of the warping memory limit, add heuristics to determine if we must split the target window in case the 'fill ratio' of the source dataset is too low (#3120)"

[Even Rouault]

Milestone renamed