Opened 16 years ago

Closed 16 years ago

#2033 closed defect (fixed)

[PATCH] gdaladdo tif_jpeg assertion failure: sp->cinfo.comm.is_decompressor

Reported by: crschmidt Owned by: warmerdam
Priority: normal Milestone: 1.5.0
Component: GDAL_Raster Version: svn-trunk
Severity: normal Keywords: jpeg geotiff
Cc: Mateusz Łoskot, dron, Even Rouault

Description

The following process seems to cause gdaladdo to blow up on this particular image:

nice -n 10 /home/crschmidt/FWTools-1.4.2/bin_safe/gdalwarp --config GDAL_CACHEMAX 500 -co TILED=yes -co BLOCKXSIZE=512 -co BLOCKYSIZE=512 -co COMPRESS=JPEG -of GTiff -t_srs EPSG:4326 n34119h8sw.tif ~/n34.tiff

~/FWTools-1.4.2/bin_safe/gdaladdo ~/n34.tiff 2 4 8 16

This is also the case with the system, 1.4.0 GDAL.

The n34.tiff, 13MB, is available from http://hypercube.telascience.org/~crschmidt/n34.tiff . Source data is from http://new.casil.ucdavis.edu/casil/remote_sensing/naip_2005/doqqs/34119/ .

Attachments (5)

openev-fwtools-2.0.1-out-tif.png (267.1 KB ) - added by Mateusz Łoskot 16 years ago.
Tested output out.tif file in OpenEV (FWTools 2.0.1, GDAL 1.5.0dev)
openev-fwtools-2.0.1-out-tif-2.png (792.4 KB ) - added by Mateusz Łoskot 16 years ago.
Zooming tested output out.tif file in OpenEV (FWTools 2.0.1, GDAL 1.5.0dev)
openev-fwtools-1.3.9-out-tif.png (30.1 KB ) - added by Mateusz Łoskot 16 years ago.
Tested output out.tif file in OpenEV (FWTools 1.3.9, GDAL 1.5.0dev - 1 month older, from 2007/10/11)
libtiff_jpeg_bug2033.patch (991 bytes ) - added by Even Rouault 16 years ago.
gdal_svn_trunk_rework_GDALDatasetCopyWholeRaster.patch (8.9 KB ) - added by Even Rouault 16 years ago.
Handle the case of big datasets and alignment of swath dimensions on the blocks of the destination dataset

Download all attachments as: .zip

Change History (17)

comment:1 by crschmidt, 16 years ago

Failure looks like this:

crschmidt@hypercube:~$ ~/FWTools-1.4.2/bin_safe/gdaladdo n34.tiff 2 4 8 16
0...10...gdaladdo: tif_jpeg.c:1552: JPEGPreEncode: Assertion `!sp->cinfo.comm.is_decompressor' failed./home/crschmidt/FWTools-1.4.2/bin_safe/gdaladdo: line 9: 17420 Aborted                 $FWTOOLS_HOME/bin/`basename $TARGET` "$@

comment:2 by Mateusz Łoskot, 16 years ago

Cc: Mateusz Łoskot added
Keywords: jpeg geotiff added

comment:3 by Mateusz Łoskot, 16 years ago

I'm not sure, but for me it looks like Chris forgot to add some steps. Here is my version of this process and I'm getting similar results bug different:

  1. gdalwarp n34119h8sw.tif out.tiff
    /dev/gdal/bugs/2033 $ gdalwarp --config GDAL_CACHEMAX 500  -co TILED=yes -co BLOCKXSIZE=512 -co BLOCKYSIZE=512  -co COMPRESS=JPEG -of GTiff -t_srs EPSG:4326 ~/data/casil/remote_sensing/naip_2005/doqqs/n34119h8sw.tif out.tiff
    GDAL: GDALOpen(/Users/mloskot/data/casil/remote_sensing/naip_2005/doqqs/n34119h8sw.tif) succeeds as GTiff.
    OGRCT: Source: +proj=utm +zone=11 +ellps=GRS80 +datum=NAD83 +units=m +no_defs 
    OGRCT: Target: +proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs 
    OGRCT: Source: +proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs 
    OGRCT: Target: +proj=utm +zone=11 +ellps=GRS80 +datum=NAD83 +units=m +no_defs 
    GDAL: GDALClose(/Users/mloskot/data/casil/remote_sensing/naip_2005/doqqs/n34119h8sw.tif)
    Creating output file that is 7474P x 7205L.
    GDAL: GDALDriver::Create(GTiff,out.tiff,7474,7205,3,Byte,0x81095a0)
    GDAL: GDALOpen(/Users/mloskot/data/casil/remote_sensing/naip_2005/doqqs/n34119h8sw.tif) succeeds as GTiff.
    Processing input file /Users/mloskot/data/casil/remote_sensing/naip_2005/doqqs/n34119h8sw.tif.
    OGRCT: Source: +proj=utm +zone=11 +ellps=GRS80 +datum=NAD83 +units=m +no_defs 
    OGRCT: Target: +proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs 
    OGRCT: Source: +proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs 
    OGRCT: Target: +proj=utm +zone=11 +ellps=GRS80 +datum=NAD83 +units=m +no_defs 
    GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
    Src=0,0,1699x3820 Dst=0,0,1868x3602
    0...10GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
    Src=1579,0,1813x3870 Dst=1868,0,1869x3602
    ...20.GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
    Src=0,3768,1580x3972 Dst=0,3602,1868x3603
    ..30..GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
    Src=1461,3819,1813x3921 Dst=1868,3602,1869x3603
    .40...50GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
    Src=3273,0,1811x3920 Dst=3737,0,1868x3602
    ...60GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
    Src=4966,0,1584x3970 Dst=5605,0,1869x3602
    ...70.GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
    Src=3156,3869,1811x3871 Dst=3737,3602,1868x3603
    ..80..GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
    Src=4849,3919,1701x3821 Dst=5605,3602,1869x3603
    .90...100 - done.
    GDAL: GDALClose(/Users/mloskot/data/casil/remote_sensing/naip_2005/doqqs/n34119h8sw.tif)
    GDAL: GDALClose(out.tiff)
    GDAL: GDALDeregister_GTiff() called.
    
  1. gdalinfo out.tiff
    ~/dev/gdal/bugs/2033 $ gdalinfo out.tiff 
    GDAL: GDALOpen(out.tiff) succeeds as GTiff.
    Driver: GTiff/GeoTIFF
    Files: out.tiff
    Size is 7474, 7205
    Coordinate System is:
    GEOGCS["WGS 84",
        DATUM["WGS_1984",
            SPHEROID["WGS 84",6378137,298.2572235630016,
                AUTHORITY["EPSG","7030"]],
            AUTHORITY["EPSG","6326"]],
        PRIMEM["Greenwich",0],
        UNIT["degree",0.0174532925199433],
        AUTHORITY["EPSG","4326"]]
    Origin = (-120.005830387315683,34.941970857729622)
    Pixel Size = (0.000009916509033,-0.000009916509033)
    Metadata:
      AREA_OR_POINT=Area
    Image Structure Metadata:
      COMPRESSION=JPEG
      INTERLEAVE=PIXEL
    OGRCT: Source: +proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs 
    OGRCT: Target: +proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs 
    Corner Coordinates:
    Upper Left  (-120.0058304,  34.9419709) (120d 0'20.99"W, 34d56'31.10"N)
    Lower Left  (-120.0058304,  34.8705224) (120d 0'20.99"W, 34d52'13.88"N)
    Upper Right (-119.9317144,  34.9419709) (119d55'54.17"W, 34d56'31.10"N)
    Lower Right (-119.9317144,  34.8705224) (119d55'54.17"W, 34d52'13.88"N)
    Center      (-119.9687724,  34.9062466) (119d58'7.58"W, 34d54'22.49"N)
    Band 1 Block=512x512 Type=Byte, ColorInterp=Red
    Band 2 Block=512x512 Type=Byte, ColorInterp=Green
    Band 3 Block=512x512 Type=Byte, ColorInterp=Blue
    GDAL: GDALClose(out.tiff)
    GDAL: GDALDeregister_GTiff() called.
    
  1. gdaladdo out.tiff 2 4 8 16
    ~/dev/gdal/bugs/2033 $ gdaladdo out.tiff 2 4 8 16
    GDAL: GDALOpen(out.tiff) succeeds as GTiff.
    0...10...20...30.tif_jpeg.c:943: failed assertion `sp->cinfo.comm.is_decompressor'
    Abort trap
    
  1. gdalinfo out.tiff

~/dev/gdal/bugs/2033 $ gdalinfo out.tiff GTiff: Opened 3737x3603 overview.

GTiff: Opened 1869x1802 overview.

GTiff: Opened 935x901 overview.

GTiff: Opened 468x451 overview.

GDAL: GDALOpen(out.tiff) succeeds as GTiff. Driver: GTiff/GeoTIFF Files: out.tiff Size is 7474, 7205 Coordinate System is: GEOGCS["WGS 84",

DATUM["WGS_1984",

SPHEROID["WGS 84",6378137,298.2572235630016,

AUTHORITY["EPSG","7030"]],

AUTHORITY["EPSG","6326"]],

PRIMEM["Greenwich",0], UNIT["degree",0.0174532925199433], AUTHORITY["EPSG","4326"]]

Origin = (-120.005830387315683,34.941970857729622) Pixel Size = (0.000009916509033,-0.000009916509033) Metadata:

AREA_OR_POINT=Area

Image Structure Metadata:

COMPRESSION=JPEG INTERLEAVE=PIXEL

OGRCT: Source: +proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs OGRCT: Target: +proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs Corner Coordinates: Upper Left (-120.0058304, 34.9419709) (120d 0'20.99"W, 34d56'31.10"N) Lower Left (-120.0058304, 34.8705224) (120d 0'20.99"W, 34d52'13.88"N) Upper Right (-119.9317144, 34.9419709) (119d55'54.17"W, 34d56'31.10"N) Lower Right (-119.9317144, 34.8705224) (119d55'54.17"W, 34d52'13.88"N) Center (-119.9687724, 34.9062466) (119d58'7.58"W, 34d54'22.49"N) Band 1 Block=512x512 Type=Byte, ColorInterp=Red

Overviews: 3737x3603, 1869x1802, 935x901, 468x451

Band 2 Block=512x512 Type=Byte, ColorInterp=Green

Overviews: 3737x3603, 1869x1802, 935x901, 468x451

Band 3 Block=512x512 Type=Byte, ColorInterp=Blue

Overviews: 3737x3603, 1869x1802, 935x901, 468x451

GDAL: GDALClose() GDAL: GDALClose() GDAL: GDALClose() GDAL: GDALClose() GDAL: GDALClose(out.tiff) GDAL: GDALDeregister_GTiff() called. }}}

  • Comparing Chris' file with mine
    ~/dev/gdal/bugs/2033 $ ls -l
    total 74288
    -rw-r--r--   1 mloskot  mloskot  19016502 Nov 28 08:50 n34.tiff
    -rw-r--r--   1 mloskot  mloskot  19016502 Nov 28 09:48 out.tiff
    

Summary:

  • The assertion is different and occurs in function JPEGPreDecode but not JPEGPreEncode
  • Inspite of the assertion catch, overviews are burned.

by Mateusz Łoskot, 16 years ago

Tested output out.tif file in OpenEV (FWTools 2.0.1, GDAL 1.5.0dev)

by Mateusz Łoskot, 16 years ago

Zooming tested output out.tif file in OpenEV (FWTools 2.0.1, GDAL 1.5.0dev)

by Mateusz Łoskot, 16 years ago

Tested output out.tif file in OpenEV (FWTools 1.3.9, GDAL 1.5.0dev - 1 month older, from 2007/10/11)

comment:4 by Mateusz Łoskot, 16 years ago

One important note, the tests I run above was made using GDAL + external libjpeg.

Here is run using internal libjpeg:

$ gdaladdo out.tiff 2 4 8 16
GTiff: Opened 3737x3603 overview.

GTiff: Opened 1869x1802 overview.

GTiff: Opened 935x901 overview.

GTiff: Opened 468x451 overview.

GDAL: GDALOpen(out.tiff) succeeds as GTiff.
0...10...tif_jpeg.c:1552: failed assertion `!sp->cinfo.comm.is_decompressor'
Abort trap

comment:5 by Mateusz Łoskot, 16 years ago

Here is a dump of my thoughts related to this problem I expressed on the #gdal:

Nov 28 14:16:32 <mloskot> there is only one place where is_decompressor is set to false
Nov 28 14:16:38 <mloskot> jcapimin.c:55
Nov 28 14:17:03 <mloskot> and execution never walks that way during processing our file
Nov 28 14:34:16 <mloskot> I'm 90% sure the problem is with uninitialized struct field. 
Nov 28 14:34:22 <mloskot> The decompression goes fine
Nov 28 14:34:55 <mloskot> but when compression of tiles starts, execution never jumps into  JPEGPreEncode so is_decompress is never initialized with FALSE
Nov 28 14:35:10 <mloskot> but this function is the only place where initialization of this member occurs
Nov 28 14:36:03 <mloskot> Unfortunately, I'm not able to understand or conclude why it does not jump into JPEGPreEncode  
Nov 28 14:39:43 <mloskot> I *suppose*, the engine uses the same handle to TIFF for decoding so for encoding, but for encoding it's still "configured" with decompressor
Nov 28 14:39:56 <mloskot> so, it's not correct for encoding in further stage of the process
Nov 28 14:40:12 <mloskot> I mean handle GTiffDataset::hTIFF and tiff tile encoder fails
Nov 28 14:41:25 <FrankW> It is possible to switch between encoding and decoding but unfortunately the encoder/decoder code for libtiff was not originally intended to do this and my efforts to support it have been not entirely comprehensive.
Nov 28 14:44:43 <mloskot> FrankW:  But gdaladdo -> gtiff driver uses the same object, comparing memory addresses of TIFF handle 
Nov 28 14:45:26 <mloskot> perhaps the gtiff driver could validate state on higher level instead of deeply in libjpeg and reconfigure if possible
Nov 28 14:45:52 <mloskot> but I'm a green in this subject and have no idea if these comments make sense at all

comment:6 by dron, 16 years ago

Cc: dron added

comment:7 by Even Rouault, 16 years ago

Cc: Even Rouault added

I'm also hitting the same bug when just translating a big image to a tiled JPEG Tiff. I've grabbed BMNG's PNG, like world.topo.bathy.200406.3x21600x21600.A2.png (http://worldwind28.arc.nasa.gov/public/500m_png) Then I do

gdal_translate -a_srs EPSG:4326 -co "COMPRESS=JPEG" -co "TILED=YES" ../../bmng/world.topo.bathy.200406.3x21600x21600.A2.png world.topo.bathy.200406.3x21600x21600.A2.tif

And I get after a few seconds:

tif_jpeg.c:945: JPEGPreDecode: Assertion `sp->cinfo.comm.is_decompressor' failed

In fact, this is more an issue with GDALDatasetCopyWholeRaster. I've recently added a check so that we don't read more than nTargetSwathSize (10 MB) data at a time. So we can get only 10 000 000 / (21 600 * 3) = 154 lines at a time. But my destination tiff is tiled, with default tile size 256 x 256. So as 154 < 256, the tiff driver must re-read already partially jpeg-encoded blocks... That's bad because it means jpeg decompression / jpeg recompression of the same block, which lowers quality. And that's bad as it hits this bug...

GDALDatasetCopyWholeRaster should be modified so that if nMemoryPerLine * nBlockYSize > nTargetSwathSize, we still grab nBlockYSize lines at a time, but not whole lines at the same time, just multiples of nBlockXSize. So that we don't have to read already encoded tiles. I hope that make sense

comment:8 by Even Rouault, 16 years ago

I'm attaching a patch that seems to fix this bug. I've been able to gdaladdo on crschmidt's dataset translated to a tiled JPEG GTiff.

However there is something I don't understand. The JPEG GTiff without overviews is 15 MB large. Once "gdaladdo 2 4 8 16" has been run, it is 31 MB large.

But if I make the original JPEG GTiff write only and run COMPRESS_OVERVIEW=JPEG gdaladdo 2 4 8 16, it produces an external geotiff that it is 7 MB large. 15 + 7 = 22 MB < 31 MB...

I'm observing the samething on my BMNG JPEG GTiff : 23 + 11 = 35 < 49 MB.

by Even Rouault, 16 years ago

Attachment: libtiff_jpeg_bug2033.patch added

comment:9 by Even Rouault, 16 years ago

Please read above : "... make the original JPEG GTiff READ-ONLY ..." and 23 + 11 = 34 ;-)

comment:10 by Even Rouault, 16 years ago

I'm attaching another patch that rework GDALDatasetCopyWholeRaster so that we still grab nBlockYSize lines in the swath, but not on the whole dataset width, in case of large datasets.

comment:11 by Even Rouault, 16 years ago

Summary: gdaladdo tif_jpeg assertion failure: sp->cinfo.comm.is_decompressor[PATCH] gdaladdo tif_jpeg assertion failure: sp->cinfo.comm.is_decompressor

by Even Rouault, 16 years ago

Handle the case of big datasets and alignment of swath dimensions on the blocks of the destination dataset

comment:12 by warmerdam, 16 years ago

Resolution: fixed
Status: newclosed

I confirmed getting the is_decompressor error when doing a gdal_translate of a wide image to a tiled jpeg compressed tiff, and then adding overviews.

I applied Even's patch with some cleanup to libtiff CVS head (libtiff 4), and then migrated that into the internal libtiff of GDAL (r13310).

I tried applying the patch in libtiff 3.9 but gdaladdo failed differently. I was unable to determine if this was due to the patch, or unrelated problems but I abandoned that effort. So libtiff 3.9 and earlier versions are likely to continue having problems with adding overviews on jpeg compressed tiff files and some other random read/write tasks (like gdalwarp).

I'm not prepared to apply the GDALDatasetCopyWholeRaster patch at this time. I feel that it is covering over problems in particular drivers, and/or just a different approach for lossily compressed formats.

Note: See TracTickets for help on using tickets.