Ticket #4771 (closed defect: fixed)

Opened 21 months ago

Last modified 20 months ago

[PATCH - libtiff] gdalwarp fails merging into deflate-compressed GTiff

Reported by: cdestigter Owned by: warmerdam
Priority: normal Milestone: 1.10.0
Component: GDAL_Raster Version:
Severity: normal Keywords: libtiff rewrite tile
Cc:

Description

gdalwarp fails when merging into this one particular deflate-compressed tiff. Other files are fine, just this specific tiff is problematic.

Images:

 https://dl.dropbox.com/u/2649727/other.tif

 https://dl.dropbox.com/u/2649727/tiled-original.tif

$ cp ~/tiled-original.tif ~/tiled-broken.tif ; LD_LIBRARY_PATH=.libs ./apps/gdalwarp ~/other.tif ~/tiled-broken.tif

Processing input file /home/cdestigter/other.tif.
Using band 4 of source image as alpha.
Using band 4 of destination image as alpha.
0...10...20...30...40...50...60...70...80...90...100 - done.
ERROR 1: ZIPDecode:Decoding error at scanline 1024, incorrect header check
ERROR 1: TIFFReadEncodedTile() failed.
ERROR 1: ZIPDecode:Decoding error at scanline 1280, incorrect header check
ERROR 1: TIFFReadEncodedTile() failed.
ERROR 1: ZIPDecode:Decoding error at scanline 1536, incorrect header check
ERROR 1: TIFFReadEncodedTile() failed.
ERROR 1: ZIPDecode:Decoding error at scanline 1792, incorrect header check
ERROR 1: TIFFReadEncodedTile() failed.

This happens on both 1.9.1 and trunk @ r24696 .

If I remove the deflate compression, it works fine:

$ cp ~/tiled-original.tif ~/tiled-broken.tif ; gdal_translate -co COMPRESS=NONE ~/tiled-broken.tif ~/tiled-uncompressed.tif ; LD_LIBRARY_PATH=.libs ./apps/gdalwarp ~/other.tif ~/tiled-uncompressed.tif

Input file size is 4096, 4096
0...10...20...30...40...50...60...70...80...90...100 - done.
Processing input file /home/cdestigter/other.tif.
Using band 4 of source image as alpha.
Using band 4 of destination image as alpha.
0...10...20...30...40...50...60...70...80...90...100 - done.

Attachments

libtiff_gdal_4771.patch Download (2.6 KB) - added by rouault 21 months ago.
test_4771.py Download (1.1 KB) - added by rouault 21 months ago.

Change History

Changed 21 months ago by rouault

This works for me with both GDAL 1.9 and trunk, with *internal* libtiff. I suspect your GDAL is built with an external libtiff, which might be a libtiff 3.x ? I believe there have been fixes in libtiff 4 to better support updating compressed tiffs.

Changed 21 months ago by cdestigter

Thanks Even. We're actually using internal libtiff:

  LIBTIFF support:           internal (BigTIFF=yes)
  LIBGEOTIFF support:        internal

Any ideas what else might be at play?

Changed 21 months ago by rouault

In fact, I do reproduce your problem. I pasted your command line without a needed adaptation (the files were not in my home directory, so I wasn't gdalwarping over the deflate-compressed tiff

Changed 21 months ago by rouault

  • keywords libtiff rewrite tile added
  • summary changed from gdalwarp fails merging into deflate-compressed GTiff to [PATCH - libtiff] gdalwarp fails merging into deflate-compressed GTiff

ok, this is another issue with update of existing tiles in compressed tiff. Attached a patch that fixes the issue, and a simple GDAL python test to reproduce the bug triggered by the above use case.

Frank, I'll let you review and apply in upstream libtiff if appropriate. This area of tile rewriting is particularly muddy ;-)

cdestigter: note that updating of compressed tiff is generally not a good idea if you are looking for the smallest possible file ( even when the operation succeeds ;-)). When the compressed updated data of a tile is larger than the original compressed data, libtiff will waste the space occupied by the original compressed data and write the new updated content at the end of the file, so you end up with "holes" in the tiff file that are never recovered. The only way to recover them is to call gdal_translate to generate a new compressed file from the file with holes.

Changed 21 months ago by rouault

Changed 21 months ago by rouault

Changed 21 months ago by warmerdam

Even, I have upstreamed the patch. Feel free to update gdal and incorporate a check in the autotest suite. Thanks!

Changed 21 months ago by rouault

  • status changed from new to closed
  • version 1.9.1 deleted
  • resolution set to fixed
  • milestone set to 2.0.0

r24777 /trunk/ (12 files in 2 dirs): Refresh internal libtiff from upstream to fix, among other things, a bug when rewriting image tiles in a deflate compressed file (#4771)

Changed 20 months ago by cdestigter

You guys are legends, thanks!

Note: See TracTickets for help on using tickets.