Opened 10 years ago

Closed 4 years ago

Last modified 4 years ago

#3120 closed enhancement (fixed)

[PATCH] gdalwarp speed-up when warping small images into a big mosaic

Reported by: peifer Owned by: warmerdam
Priority: normal Milestone: 2.0.0
Component: Algorithms Version: unspecified
Severity: normal Keywords: warp
Cc: Markus Neteler, Even Rouault

Description

Here a summary of some discussion at gdal-dev@…

Adam Nowacki wrote:
Some rather counterintuitive gdalwarp behavior: the bigger dfWarpMemoryLimit (-wm setting) the more cpu time will be wasted on warping not existing pixels. Why? Warping begins with the destination window size of entire output image size. If this size is larger than dfWarpMemoryLimit it is split in half along the longest axis and any half that doesn't contain the currently processed source file is discarded. With large dfWarpMemoryLimit this subdivision process will stop early with still large portions of out of source image pixels.

My observation while gdalwarping *many* small GeoTiffs? into a big mosaic, with -wm 300:

Src=0,0,3601x3601 Dst=36375,31125,12125x10375

I understand this as follows: the source image of 3601x3601 pixels is read as one chunk (which after reprojection and resampling should be around 800x1100 pixels). However, 12125x10375 pixels are actually written to the output file. So 99.5% of the destination window must be completely unrelated to my input image.

If I counter-intuitively reduce the memory limit to, say: 40 MB and leave anything else unchanged, then gdalwarp is processing the input files much faster.

Perhaps gdalwarp should not only test if the destination window fits into memory, but also check what would be the minimum destination window for warping the input image. This could speed up the mosaicing of small tiles into a bigger output file.

Adam provided a patch with a new variable dfFillRatio, which made things even faster. The patch is here: http://pastebin.com/m45e46f53

Attachments (1)

ticket_3120.patch (2.9 KB) - added by Even Rouault 10 years ago.
Patch from http://pastebin.com/m45e46f53

Download all attachments as: .zip

Change History (8)

Changed 10 years ago by Even Rouault

Attachment: ticket_3120.patch added

comment:1 Changed 10 years ago by Even Rouault

Component: GDAL_RasterAlgorithms
Summary: Strange things with gdalwarp ...[PATCH] gdalwarp speed-up when warping small images into a big mosaic

comment:2 Changed 10 years ago by Markus Neteler

Cc: Markus Neteler added

comment:3 Changed 9 years ago by peifer

Cc: Even Rouault added

Even,

(Frank speaking here with Hermann)

If you are confident about this change, just go ahead and apply it.

comment:4 Changed 4 years ago by Jukka Rahkonen

Patch not applied yet. Recent mailing list thread shows that problem with counterintuitive behaviour still exist http://thread.gmane.org/gmane.comp.gis.gdal.devel/39250.

comment:5 Changed 4 years ago by peifer

IIRC: At the time (in other words: 5 years ago) I applied the patch locally and it was working fine. I never understood why it has been ignored.

comment:6 in reply to:  5 Changed 4 years ago by Even Rouault

Keywords: warp added
Milestone: 2.0
Resolution: fixed
Status: newclosed

Replying to peifer:

IIRC: At the time (in other words: 5 years ago) I applied the patch locally and it was working fine. I never understood why it has been ignored.

Well, I spent more than 2 hours of unfunded time reviewing, testing, fixing/improving the initial patch and adding a regression test for that issue... If there are issues you'd wish to be fixed more quickly, there are developers who can be sponsored for that.

trunk r28258 "warper: regardless of the warping memory limit, add heuristics to determine if we must split the target window in case the 'fill ratio' of the source dataset is too low (#3120)"

comment:7 Changed 4 years ago by Even Rouault

Milestone: 2.02.0.0

Milestone renamed

Note: See TracTickets for help on using tickets.