Ticket #2033 (closed defect: fixed)

Opened 7 months ago

Last modified 7 months ago

[PATCH] gdaladdo tif_jpeg assertion failure: sp->cinfo.comm.is_decompressor

Reported by: crschmidt Assigned to: warmerdam
Priority: normal Milestone: 1.5.0
Component: GDAL_Raster Version: svn-trunk
Severity: normal Keywords: jpeg geotiff
Cc: mloskot, dron, rouault

Description

The following process seems to cause gdaladdo to blow up on this particular image:

nice -n 10 /home/crschmidt/FWTools-1.4.2/bin_safe/gdalwarp --config GDAL_CACHEMAX 500 -co TILED=yes -co BLOCKXSIZE=512 -co BLOCKYSIZE=512 -co COMPRESS=JPEG -of GTiff -t_srs EPSG:4326 n34119h8sw.tif ~/n34.tiff

~/FWTools-1.4.2/bin_safe/gdaladdo ~/n34.tiff 2 4 8 16

This is also the case with the system, 1.4.0 GDAL.

The n34.tiff, 13MB, is available from http://hypercube.telascience.org/~crschmidt/n34.tiff . Source data is from http://new.casil.ucdavis.edu/casil/remote_sensing/naip_2005/doqqs/34119/ .

Attachments

openev-fwtools-2.0.1-out-tif.png (267.1 kB) - added by mloskot on 11/28/07 04:06:01.
Tested output out.tif file in OpenEV (FWTools 2.0.1, GDAL 1.5.0dev)
openev-fwtools-2.0.1-out-tif-2.png (0.8 MB) - added by mloskot on 11/28/07 04:07:20.
Zooming tested output out.tif file in OpenEV (FWTools 2.0.1, GDAL 1.5.0dev)
openev-fwtools-1.3.9-out-tif.png (30.1 kB) - added by mloskot on 11/28/07 04:08:47.
Tested output out.tif file in OpenEV (FWTools 1.3.9, GDAL 1.5.0dev - 1 month older, from 2007/10/11)
libtiff_jpeg_bug2033.patch (1.0 kB) - added by rouault on 12/09/07 11:11:43.
gdal_svn_trunk_rework_GDALDatasetCopyWholeRaster.patch (8.9 kB) - added by rouault on 12/10/07 15:26:01.
Handle the case of big datasets and alignment of swath dimensions on the blocks of the destination dataset

Change History

11/28/07 02:25:59 changed by crschmidt

Failure looks like this: 

crschmidt@hypercube:~$ ~/FWTools-1.4.2/bin_safe/gdaladdo n34.tiff 2 4 8 16
0...10...gdaladdo: tif_jpeg.c:1552: JPEGPreEncode: Assertion `!sp->cinfo.comm.is_decompressor' failed./home/crschmidt/FWTools-1.4.2/bin_safe/gdaladdo: line 9: 17420 Aborted                 $FWTOOLS_HOME/bin/`basename $TARGET` "$@

11/28/07 02:54:18 changed by mloskot

  • keywords set to jpeg geotiff.
  • cc set to mloskot.

11/28/07 04:01:37 changed by mloskot

I'm not sure, but for me it looks like Chris forgot to add some steps. Here is my version of this process and I'm getting similar results bug different:

  1. gdalwarp n34119h8sw.tif out.tiff 
    /dev/gdal/bugs/2033 $ gdalwarp --config GDAL_CACHEMAX 500  -co TILED=yes -co BLOCKXSIZE=512 -co BLOCKYSIZE=512  -co COMPRESS=JPEG -of GTiff -t_srs EPSG:4326 ~/data/casil/remote_sensing/naip_2005/doqqs/n34119h8sw.tif out.tiff
GDAL: GDALOpen(/Users/mloskot/data/casil/remote_sensing/naip_2005/doqqs/n34119h8sw.tif) succeeds as GTiff.
OGRCT: Source: +proj=utm +zone=11 +ellps=GRS80 +datum=NAD83 +units=m +no_defs 
OGRCT: Target: +proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs 
OGRCT: Source: +proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs 
OGRCT: Target: +proj=utm +zone=11 +ellps=GRS80 +datum=NAD83 +units=m +no_defs 
GDAL: GDALClose(/Users/mloskot/data/casil/remote_sensing/naip_2005/doqqs/n34119h8sw.tif)
Creating output file that is 7474P x 7205L.
GDAL: GDALDriver::Create(GTiff,out.tiff,7474,7205,3,Byte,0x81095a0)
GDAL: GDALOpen(/Users/mloskot/data/casil/remote_sensing/naip_2005/doqqs/n34119h8sw.tif) succeeds as GTiff.
Processing input file /Users/mloskot/data/casil/remote_sensing/naip_2005/doqqs/n34119h8sw.tif.
OGRCT: Source: +proj=utm +zone=11 +ellps=GRS80 +datum=NAD83 +units=m +no_defs 
OGRCT: Target: +proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs 
OGRCT: Source: +proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs 
OGRCT: Target: +proj=utm +zone=11 +ellps=GRS80 +datum=NAD83 +units=m +no_defs 
GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=0,0,1699x3820 Dst=0,0,1868x3602
0...10GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=1579,0,1813x3870 Dst=1868,0,1869x3602
...20.GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=0,3768,1580x3972 Dst=0,3602,1868x3603
..30..GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=1461,3819,1813x3921 Dst=1868,3602,1869x3603
.40...50GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=3273,0,1811x3920 Dst=3737,0,1868x3602
...60GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=4966,0,1584x3970 Dst=5605,0,1869x3602
...70.GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=3156,3869,1811x3871 Dst=3737,3602,1868x3603
..80..GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=4849,3919,1701x3821 Dst=5605,3602,1869x3603
.90...100 - done.
GDAL: GDALClose(/Users/mloskot/data/casil/remote_sensing/naip_2005/doqqs/n34119h8sw.tif)
GDAL: GDALClose(out.tiff)
GDAL: GDALDeregister_GTiff() called.
  1. gdalinfo out.tiff 
    ~/dev/gdal/bugs/2033 $ gdalinfo out.tiff 
GDAL: GDALOpen(out.tiff) succeeds as GTiff.
Driver: GTiff/GeoTIFF
Files: out.tiff
Size is 7474, 7205
Coordinate System is:
GEOGCS["WGS 84",
    DATUM["WGS_1984",
        SPHEROID["WGS 84",6378137,298.2572235630016,
            AUTHORITY["EPSG","7030"]],
        AUTHORITY["EPSG","6326"]],
    PRIMEM["Greenwich",0],
    UNIT["degree",0.0174532925199433],
    AUTHORITY["EPSG","4326"]]
Origin = (-120.005830387315683,34.941970857729622)
Pixel Size = (0.000009916509033,-0.000009916509033)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  COMPRESSION=JPEG
  INTERLEAVE=PIXEL
OGRCT: Source: +proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs 
OGRCT: Target: +proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs 
Corner Coordinates:
Upper Left  (-120.0058304,  34.9419709) (120d 0'20.99"W, 34d56'31.10"N)
Lower Left  (-120.0058304,  34.8705224) (120d 0'20.99"W, 34d52'13.88"N)
Upper Right (-119.9317144,  34.9419709) (119d55'54.17"W, 34d56'31.10"N)
Lower Right (-119.9317144,  34.8705224) (119d55'54.17"W, 34d52'13.88"N)
Center      (-119.9687724,  34.9062466) (119d58'7.58"W, 34d54'22.49"N)
Band 1 Block=512x512 Type=Byte, ColorInterp=Red
Band 2 Block=512x512 Type=Byte, ColorInterp=Green
Band 3 Block=512x512 Type=Byte, ColorInterp=Blue
GDAL: GDALClose(out.tiff)
GDAL: GDALDeregister_GTiff() called.
  1. gdaladdo out.tiff 2 4 8 16 
    ~/dev/gdal/bugs/2033 $ gdaladdo out.tiff 2 4 8 16
GDAL: GDALOpen(out.tiff) succeeds as GTiff.
0...10...20...30.tif_jpeg.c:943: failed assertion `sp->cinfo.comm.is_decompressor'
Abort trap
  1. gdalinfo out.tiff

~/dev/gdal/bugs/2033 $ gdalinfo out.tiff GTiff: Opened 3737x3603 overview.

GTiff: Opened 1869x1802 overview.

GTiff: Opened 935x901 overview.

GTiff: Opened 468x451 overview.

GDAL: GDALOpen(out.tiff) succeeds as GTiff. Driver: GTiff/GeoTIFF Files: out.tiff Size is 7474, 7205 Coordinate System is: GEOGCS["WGS 84",

DATUM["WGS_1984",

SPHEROID["WGS 84",6378137,298.2572235630016,

AUTHORITY["EPSG","7030"]],

AUTHORITY["EPSG","6326"]],

PRIMEM["Greenwich",0], UNIT["degree",0.0174532925199433], AUTHORITY["EPSG","4326"]]

Origin = (-120.005830387315683,34.941970857729622) Pixel Size = (0.000009916509033,-0.000009916509033) Metadata:

AREA_OR_POINT=Area

Image Structure Metadata:

COMPRESSION=JPEG INTERLEAVE=PIXEL

OGRCT: Source: +proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs OGRCT: Target: +proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs Corner Coordinates: Upper Left (-120.0058304, 34.9419709) (120d 0'20.99"W, 34d56'31.10"N) Lower Left (-120.0058304, 34.8705224) (120d 0'20.99"W, 34d52'13.88"N) Upper Right (-119.9317144, 34.9419709) (119d55'54.17"W, 34d56'31.10"N) Lower Right (-119.9317144, 34.8705224) (119d55'54.17"W, 34d52'13.88"N) Center (-119.9687724, 34.9062466) (119d58'7.58"W, 34d54'22.49"N) Band 1 Block=512x512 Type=Byte, ColorInterp?=Red

Overviews: 3737x3603, 1869x1802, 935x901, 468x451

Band 2 Block=512x512 Type=Byte, ColorInterp?=Green

Overviews: 3737x3603, 1869x1802, 935x901, 468x451

Band 3 Block=512x512 Type=Byte, ColorInterp?=Blue

Overviews: 3737x3603, 1869x1802, 935x901, 468x451

GDAL: GDALClose() GDAL: GDALClose() GDAL: GDALClose() GDAL: GDALClose() GDAL: GDALClose(out.tiff) GDAL: GDALDeregister_GTiff() called. }}}

  • Comparing Chris' file with mine 
    ~/dev/gdal/bugs/2033 $ ls -l
total 74288
-rw-r--r--   1 mloskot  mloskot  19016502 Nov 28 08:50 n34.tiff
-rw-r--r--   1 mloskot  mloskot  19016502 Nov 28 09:48 out.tiff

Summary:

  • The assertion is different and occurs in function JPEGPreDecode but not JPEGPreEncode
  • Inspite of the assertion catch, overviews are burned.

11/28/07 04:06:01 changed by mloskot

  • attachment openev-fwtools-2.0.1-out-tif.png added.

Tested output out.tif file in OpenEV (FWTools 2.0.1, GDAL 1.5.0dev)

11/28/07 04:07:20 changed by mloskot

  • attachment openev-fwtools-2.0.1-out-tif-2.png added.

Zooming tested output out.tif file in OpenEV (FWTools 2.0.1, GDAL 1.5.0dev)

11/28/07 04:08:47 changed by mloskot

  • attachment openev-fwtools-1.3.9-out-tif.png added.

Tested output out.tif file in OpenEV (FWTools 1.3.9, GDAL 1.5.0dev - 1 month older, from 2007/10/11)

11/28/07 08:28:48 changed by mloskot

One important note, the tests I run above was made using GDAL + external libjpeg.

Here is run using internal libjpeg: 

$ gdaladdo out.tiff 2 4 8 16
GTiff: Opened 3737x3603 overview.

GTiff: Opened 1869x1802 overview.

GTiff: Opened 935x901 overview.

GTiff: Opened 468x451 overview.

GDAL: GDALOpen(out.tiff) succeeds as GTiff.
0...10...tif_jpeg.c:1552: failed assertion `!sp->cinfo.comm.is_decompressor'
Abort trap

11/28/07 12:49:49 changed by mloskot

Here is a dump of my thoughts related to this problem I expressed on the #gdal: 

Nov 28 14:16:32 <mloskot> there is only one place where is_decompressor is set to false
Nov 28 14:16:38 <mloskot> jcapimin.c:55
Nov 28 14:17:03 <mloskot> and execution never walks that way during processing our file
Nov 28 14:34:16 <mloskot> I'm 90% sure the problem is with uninitialized struct field. 
Nov 28 14:34:22 <mloskot> The decompression goes fine
Nov 28 14:34:55 <mloskot> but when compression of tiles starts, execution never jumps into  JPEGPreEncode so is_decompress is never initialized with FALSE
Nov 28 14:35:10 <mloskot> but this function is the only place where initialization of this member occurs
Nov 28 14:36:03 <mloskot> Unfortunately, I'm not able to understand or conclude why it does not jump into JPEGPreEncode  
Nov 28 14:39:43 <mloskot> I *suppose*, the engine uses the same handle to TIFF for decoding so for encoding, but for encoding it's still "configured" with decompressor
Nov 28 14:39:56 <mloskot> so, it's not correct for encoding in further stage of the process
Nov 28 14:40:12 <mloskot> I mean handle GTiffDataset::hTIFF and tiff tile encoder fails
Nov 28 14:41:25 <FrankW> It is possible to switch between encoding and decoding but unfortunately the encoder/decoder code for libtiff was not originally intended to do this and my efforts to support it have been not entirely comprehensive.
Nov 28 14:44:43 <mloskot> FrankW:  But gdaladdo -> gtiff driver uses the same object, comparing memory addresses of TIFF handle 
Nov 28 14:45:26 <mloskot> perhaps the gtiff driver could validate state on higher level instead of deeply in libjpeg and reconfigure if possible
Nov 28 14:45:52 <mloskot> but I'm a green in this subject and have no idea if these comments make sense at all

11/28/07 12:55:48 changed by dron

  • cc changed from mloskot to mloskot, dron.

12/09/07 09:53:02 changed by rouault

  • cc changed from mloskot, dron to mloskot, dron, rouault.

I'm also hitting the same bug when just translating a big image to a tiled JPEG Tiff. I've grabbed BMNG's PNG, like world.topo.bathy.200406.3x21600x21600.A2.png (http://worldwind28.arc.nasa.gov/public/500m_png) Then I do 

gdal_translate -a_srs EPSG:4326 -co "COMPRESS=JPEG" -co "TILED=YES" ../../bmng/world.topo.bathy.200406.3x21600x21600.A2.png world.topo.bathy.200406.3x21600x21600.A2.tif

And I get after a few seconds: 

tif_jpeg.c:945: JPEGPreDecode: Assertion `sp->cinfo.comm.is_decompressor' failed

In fact, this is more an issue with GDALDatasetCopyWholeRaster. I've recently added a check so that we don't read more than nTargetSwathSize (10 MB) data at a time. So we can get only 10 000 000 / (21 600 * 3) = 154 lines at a time. But my destination tiff is tiled, with default tile size 256 x 256. So as 154 < 256, the tiff driver must re-read already partially jpeg-encoded blocks... That's bad because it means jpeg decompression / jpeg recompression of the same block, which lowers quality. And that's bad as it hits this bug...

GDALDatasetCopyWholeRaster should be modified so that if nMemoryPerLine * nBlockYSize > nTargetSwathSize, we still grab nBlockYSize lines at a time, but not whole lines at the same time, just multiples of nBlockXSize. So that we don't have to read already encoded tiles. I hope that make sense

12/09/07 11:11:14 changed by rouault

I'm attaching a patch that seems to fix this bug. I've been able to gdaladdo on crschmidt's dataset translated to a tiled JPEG GTiff.

However there is something I don't understand. The JPEG GTiff without overviews is 15 MB large. Once "gdaladdo 2 4 8 16" has been run, it is 31 MB large.

But if I make the original JPEG GTiff write only and run COMPRESS_OVERVIEW=JPEG gdaladdo 2 4 8 16, it produces an external geotiff that it is 7 MB large. 15 + 7 = 22 MB < 31 MB...

I'm observing the samething on my BMNG JPEG GTiff : 23 + 11 = 35 < 49 MB.

12/09/07 11:11:43 changed by rouault

  • attachment libtiff_jpeg_bug2033.patch added.

12/09/07 11:28:11 changed by rouault

Please read above : "... make the original JPEG GTiff READ-ONLY ..." and 23 + 11 = 34 ;-)

12/09/07 19:00:22 changed by rouault

I'm attaching another patch that rework GDALDatasetCopyWholeRaster so that we still grab nBlockYSize lines in the swath, but not on the whole dataset width, in case of large datasets.

12/10/07 14:01:42 changed by rouault

  • summary changed from gdaladdo tif_jpeg assertion failure: sp->cinfo.comm.is_decompressor to [PATCH] gdaladdo tif_jpeg assertion failure: sp->cinfo.comm.is_decompressor.

12/10/07 15:26:01 changed by rouault

  • attachment gdal_svn_trunk_rework_GDALDatasetCopyWholeRaster.patch added.

Handle the case of big datasets and alignment of swath dimensions on the blocks of the destination dataset

12/10/07 17:21:41 changed by warmerdam

  • status changed from new to closed.
  • resolution set to fixed.

I confirmed getting the is_decompressor error when doing a gdal_translate of a wide image to a tiled jpeg compressed tiff, and then adding overviews.

I applied Even's patch with some cleanup to libtiff CVS head (libtiff 4), and then migrated that into the internal libtiff of GDAL (r13310).

I tried applying the patch in libtiff 3.9 but gdaladdo failed differently. I was unable to determine if this was due to the patch, or unrelated problems but I abandoned that effort. So libtiff 3.9 and earlier versions are likely to continue having problems with adding overviews on jpeg compressed tiff files and some other random read/write tasks (like gdalwarp).

I'm not prepared to apply the GDALDatasetCopyWholeRaster patch at this time. I feel that it is covering over problems in particular drivers, and/or just a different approach for lossily compressed formats.