[PATCH - libtiff] gdalwarp fails merging into deflate-compressed GTiff

[cdestigter]

gdalwarp fails when merging into this one particular deflate-compressed tiff. Other files are fine, just this specific tiff is problematic.

Images:

​https://dl.dropbox.com/u/2649727/other.tif

​https://dl.dropbox.com/u/2649727/tiled-original.tif

$ cp ~/tiled-original.tif ~/tiled-broken.tif ; LD_LIBRARY_PATH=.libs ./apps/gdalwarp ~/other.tif ~/tiled-broken.tif

Processing input file /home/cdestigter/other.tif.
Using band 4 of source image as alpha.
Using band 4 of destination image as alpha.
0...10...20...30...40...50...60...70...80...90...100 - done.
ERROR 1: ZIPDecode:Decoding error at scanline 1024, incorrect header check
ERROR 1: TIFFReadEncodedTile() failed.
ERROR 1: ZIPDecode:Decoding error at scanline 1280, incorrect header check
ERROR 1: TIFFReadEncodedTile() failed.
ERROR 1: ZIPDecode:Decoding error at scanline 1536, incorrect header check
ERROR 1: TIFFReadEncodedTile() failed.
ERROR 1: ZIPDecode:Decoding error at scanline 1792, incorrect header check
ERROR 1: TIFFReadEncodedTile() failed.

This happens on both 1.9.1 and trunk @ r24696 .

If I remove the deflate compression, it works fine:

$ cp ~/tiled-original.tif ~/tiled-broken.tif ; gdal_translate -co COMPRESS=NONE ~/tiled-broken.tif ~/tiled-uncompressed.tif ; LD_LIBRARY_PATH=.libs ./apps/gdalwarp ~/other.tif ~/tiled-uncompressed.tif

Input file size is 4096, 4096
0...10...20...30...40...50...60...70...80...90...100 - done.
Processing input file /home/cdestigter/other.tif.
Using band 4 of source image as alpha.
Using band 4 of destination image as alpha.
0...10...20...30...40...50...60...70...80...90...100 - done.

[Even Rouault]

This works for me with both GDAL 1.9 and trunk, with *internal* libtiff. I suspect your GDAL is built with an external libtiff, which might be a libtiff 3.x ? I believe there have been fixes in libtiff 4 to better support updating compressed tiffs.

[cdestigter]

Thanks Even. We're actually using internal libtiff:

  LIBTIFF support:           internal (BigTIFF=yes)
  LIBGEOTIFF support:        internal

Any ideas what else might be at play?

[Even Rouault]

In fact, I do reproduce your problem. I pasted your command line without a needed adaptation (the files were not in my home directory, so I wasn't gdalwarping over the deflate-compressed tiff

[Even Rouault]

ok, this is another issue with update of existing tiles in compressed tiff. Attached a patch that fixes the issue, and a simple GDAL python test to reproduce the bug triggered by the above use case.

Frank, I'll let you review and apply in upstream libtiff if appropriate. This area of tile rewriting is particularly muddy ;-)

cdestigter: note that updating of compressed tiff is generally not a good idea if you are looking for the smallest possible file ( even when the operation succeeds ;-)). When the compressed updated data of a tile is larger than the original compressed data, libtiff will waste the space occupied by the original compressed data and write the new updated content at the end of the file, so you end up with "holes" in the tiff file that are never recovered. The only way to recover them is to call gdal_translate to generate a new compressed file from the file with holes.

[warmerdam]

Even, I have upstreamed the patch. Feel free to update gdal and incorporate a check in the autotest suite. Thanks!

[Even Rouault]

r24777 /trunk/ (12 files in 2 dirs): Refresh internal libtiff from upstream to fix, among other things, a bug when rewriting image tiles in a deflate compressed file (#4771)

[cdestigter]

You guys are legends, thanks!