Opened 13 years ago

Closed 13 years ago

#3894 closed defect (fixed)

Certain JPEG compressed TIFF does not work with GDAL

Reported by: rprinceley Owned by: warmerdam
Priority: normal Milestone: 1.8.1
Component: GDAL_Raster Version: svn-trunk
Severity: normal Keywords: gtiff jpeg
Cc: gaopeng, warmerdam

Description (last modified by rprinceley)

Attached TIFF file don't work with GDAL (trunk). Decoding fails after a few rows returning the remaining as gray.

gdal_translate reports:

Input file size is 5840, 4188
0Warning 1: JPEGLib:Corrupt JPEG data: premature end of data segment
...10...20...30...40...50...60...70...80...90..ERROR 1: JPEGLib:Unsupported marker type 0x4e
.100 - done.

I'm not sure if the files are encoded incorrectly or corrupt, but other applications (non libtiff) have no problem opening them.

Adding link (can't attach, exceeds 1MB), file is here: http://dl.dropbox.com/u/18026572/0064E.tif

Attachments (1)

tif_lzma.c.3894.patch (731 bytes ) - added by Even Rouault 13 years ago.

Download all attachments as: .zip

Change History (10)

comment:1 by Even Rouault, 13 years ago

You forgot to attach the file. And which libtiff is used by your GDAL version : internal libtiff or system libtiff ? If system libtiff, which version is it ?

comment:2 by rprinceley, 13 years ago

Description: modified (diff)

in reply to:  1 comment:3 by rprinceley, 13 years ago

Replying to rouault:

You forgot to attach the file. And which libtiff is used by your GDAL version : internal libtiff or system libtiff ? If system libtiff, which version is it ?

I'm using internal libtiff.

comment:4 by Even Rouault, 13 years ago

Cc: warmerdam added
Milestone: 1.8.0

Hum, I also reproduce the failure with this image and the one attached to #3259

It works OK with 1.7 branch.

I've identified the failure to have begun with the introduction of CHUNKY_STRIP_READ_SUPPORT capability in libtiff (#3514). If I disable it, I can read the images successfully.

I've added a test that triggers the error in r21406 (by using the image attached to #3259). What is interesting is that the libjpeg error message from one run to another is not the same, so apparently there is likely some undefined behaviour due to not enough data feed to libtiff or undefined/random data passed to it (Valgrind would tell...)

I've tried to increase the "read_ahead = tif->tif_scanlinesize * 2 + 5000" to "read_ahead = tif->tif_scanlinesize * 16 + 5000", but it didn't help and I even got crashes when reading http://dl.dropbox.com/u/18026572/0064E.tif.

I think I'm going to let Frank investigate this as CHUNKY_STRIP_READ_SUPPORT is his baby ;-)

comment:5 by warmerdam, 13 years ago

Keywords: gtiff jpeg added
Status: newassigned

comment:6 by warmerdam, 13 years ago

the core problem was that the TIFF jpeg deocoder (tif_jpeg.c) was not maintaining tif_rawcp and tif_rawcc which the chunky strip filler depends on to know how much data the algorithm has consumed.

I have added this capability in CVS head (4.x) of libtiff.

I also increased the read-ahead since jpeg compression, with 16x16 tiles, might need up to 16 scanilnes worth of data in the input buffer to read a scanline of data. However, that did not appear to be strictly needed for this image.

I have not yet migrated the changes into libtiff. I would first like to review other TIFF compression formats to see if they have a similar issue. Some definately do not.

comment:7 by warmerdam, 13 years ago

Resolution: fixed
Status: assignedclosed

It appears the PixarLog compression format has a similar problem - I have not fixed it since the format is not widely used. I did add a note about it in the code.

The tif_zip (AdobeDeflate) code also had the problem. I have corrected this in libtiff CVS HEAD.

I have refreshed GDAL trunk's libtiff from upstream to get the fixes (r21418).

I have backported the essential changes to the 1.6-esri branch (r21419), including a fix for the scanline size that might have widely use for jpeg compressed images but was certainly necessary for this image.

I believe everything related to this ticket is now addressed.

comment:8 by Even Rouault, 13 years ago

Resolution: fixed
Status: closedreopened

I've added in r21422 a test that triggers the issues of that ticket quite reliably on a libtiff without your latest corrections. I added LZMA in the list and it is affected by the bug. Luckily as LZMA is a copy&paste of ZIP, your patch for tif_zip.c applies nicely for tif_lzma.c. See attached patch

by Even Rouault, 13 years ago

Attachment: tif_lzma.c.3894.patch added

comment:9 by Even Rouault, 13 years ago

Resolution: fixed
Status: reopenedclosed

r21890 /trunk/gdal/frmts/gtiff/libtiff/ (tif_getimage.c tif_jpeg.c tif_lzma.c): refresh internal libtiff (fixes lzma and CHUNKY_STRIP_READ_SUPPORT support, #3894)

Note: See TracTickets for help on using tickets.