gdalwarp very slow at updating big rasters, even if a small part is changed

[redl]

Hi,

gdalwarp is very slow when the output is a big already existing raster, even if only a small part has to be updated (by providing a cutline).

gdalwarp -cutline C:/cutline.csv C:/source.tif C:/target.tif

This takes over 45 seconds to process. To compare, copying the target.tif through gdal_translate takes around 30 seconds.

If I crop target.tif to the small part affected by the cutline, this executes in less than 1 sec. So it's clearly not the actual wrapping that is slow, but there's something in the save method.

It seems it's rewriting the whole raster, where as this is obviously unnecessary in case of updating an existing raster !

Thanks

Here's target.tif's infos using gdalinfo :

Driver: GTiff/GeoTIFF
Files: C:\fusionDTK.tif
Size is 16667, 12500
Coordinate System is:
PROJCS["WGS 84 / UTM zone 28N",
    GEOGCS["WGS 84",
        DATUM["WGS_1984",
            SPHEROID["WGS 84",6378137,298.257223563,
                AUTHORITY["EPSG","7030"]],
            AUTHORITY["EPSG","6326"]],
        PRIMEM["Greenwich",0],
        UNIT["degree",0.0174532925199433],
        AUTHORITY["EPSG","4326"]],
    PROJECTION["Transverse_Mercator"],
    PARAMETER["latitude_of_origin",0],
    PARAMETER["central_meridian",-15],
    PARAMETER["scale_factor",0.9996],
    PARAMETER["false_easting",500000],
    PARAMETER["false_northing",0],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    AUTHORITY["EPSG","32628"]]
Origin = (240000.000000000000000,1637500.000000000000000)
Pixel Size = (0.600000000000000,-0.600000000000000)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  (  240000.000, 1637500.000) ( 17d24'56.05"W, 14d47'56.55"N)
Lower Left  (  240000.000, 1630000.000) ( 17d24'53.35"W, 14d43'52.65"N)
Upper Right (  250000.200, 1637500.000) ( 17d19'21.78"W, 14d47'59.99"N)
Lower Right (  250000.200, 1630000.000) ( 17d19'19.18"W, 14d43'56.08"N)
Center      (  245000.100, 1633750.000) ( 17d22' 7.59"W, 14d45'56.34"N)
Band 1 Block=16667x1 Type=Byte, ColorInterp=Red
Band 2 Block=16667x1 Type=Byte, ColorInterp=Green
Band 3 Block=16667x1 Type=Byte, ColorInterp=Blue

[Jukka Rahkonen]

Could you add gdalinfo of target.tif? Just to make sure that it is not compressed.

[redl]

Ok I updated the ticket.

[Even Rouault]

By default, gdalwarp will indeed rewrite all the target dataset, even in zones where there is nothing to change, unless you specify "-wo SKIP_NOSOURCE=YES" (see doc in ​http://www.gdal.org/structGDALWarpOptions.html#a0ed77f9917bb96c7a9aabd73d4d06e08 )

*However*... if you specify it in your use case, it will only consider the extent of source.tif and not the extent of the cutline, so if source.tif and target.tif have the same extent, this will be of no use.

It would indeed be nice to have "-wo SKIP_NOSOURCE=YES" takes the cutline into account (by taking into account if the cutline blend distance is specified).

And also have SKIP_NOSOURCE=YES to be the default behaviour in situations where it is generally not risky to do so (when updating an existing dataset is should be OK, unless the target dataset is a sparse GeoTIFF, in which case you might want it be completely initialized. But that could be argued that this is a particular case, and that in that case the user might also explicitly set SKIP_NOSOURCE=YES).

[Jukka Rahkonen]

Am I right when I suppose that for updating a tiff that is written row-by-row (Block=16667x1) the affected rows must be rewritten entirely? And that tiled tiff suits better for on-place updates?

[Even Rouault]

Replying to jratike80:

Am I right when I suppose that for updating a tiff that is written row-by-row (Block=16667x1) the affected rows must be rewritten entirely? And that tiled tiff suits better for on-place updates?

Yes, you're right on that too.

[redl]

Awesome, this fixes my problem! Thank you rouault !

The function is very fast now, it doesn't seem the issue raised by jratike80 has much impact on the performance.

[Even Rouault]

Closing