Opened 16 years ago
Closed 16 years ago
#2033 closed defect (fixed)
[PATCH] gdaladdo tif_jpeg assertion failure: sp->cinfo.comm.is_decompressor
Reported by: | crschmidt | Owned by: | warmerdam |
---|---|---|---|
Priority: | normal | Milestone: | 1.5.0 |
Component: | GDAL_Raster | Version: | svn-trunk |
Severity: | normal | Keywords: | jpeg geotiff |
Cc: | Mateusz Łoskot, dron, Even Rouault |
Description
The following process seems to cause gdaladdo to blow up on this particular image:
nice -n 10 /home/crschmidt/FWTools-1.4.2/bin_safe/gdalwarp --config GDAL_CACHEMAX 500 -co TILED=yes -co BLOCKXSIZE=512 -co BLOCKYSIZE=512 -co COMPRESS=JPEG -of GTiff -t_srs EPSG:4326 n34119h8sw.tif ~/n34.tiff
~/FWTools-1.4.2/bin_safe/gdaladdo ~/n34.tiff 2 4 8 16
This is also the case with the system, 1.4.0 GDAL.
The n34.tiff, 13MB, is available from http://hypercube.telascience.org/~crschmidt/n34.tiff . Source data is from http://new.casil.ucdavis.edu/casil/remote_sensing/naip_2005/doqqs/34119/ .
Attachments (5)
Change History (17)
comment:1 by , 16 years ago
comment:2 by , 16 years ago
Cc: | added |
---|---|
Keywords: | jpeg geotiff added |
comment:3 by , 16 years ago
I'm not sure, but for me it looks like Chris forgot to add some steps. Here is my version of this process and I'm getting similar results bug different:
- gdalwarp n34119h8sw.tif out.tiff
/dev/gdal/bugs/2033 $ gdalwarp --config GDAL_CACHEMAX 500 -co TILED=yes -co BLOCKXSIZE=512 -co BLOCKYSIZE=512 -co COMPRESS=JPEG -of GTiff -t_srs EPSG:4326 ~/data/casil/remote_sensing/naip_2005/doqqs/n34119h8sw.tif out.tiff GDAL: GDALOpen(/Users/mloskot/data/casil/remote_sensing/naip_2005/doqqs/n34119h8sw.tif) succeeds as GTiff. OGRCT: Source: +proj=utm +zone=11 +ellps=GRS80 +datum=NAD83 +units=m +no_defs OGRCT: Target: +proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs OGRCT: Source: +proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs OGRCT: Target: +proj=utm +zone=11 +ellps=GRS80 +datum=NAD83 +units=m +no_defs GDAL: GDALClose(/Users/mloskot/data/casil/remote_sensing/naip_2005/doqqs/n34119h8sw.tif) Creating output file that is 7474P x 7205L. GDAL: GDALDriver::Create(GTiff,out.tiff,7474,7205,3,Byte,0x81095a0) GDAL: GDALOpen(/Users/mloskot/data/casil/remote_sensing/naip_2005/doqqs/n34119h8sw.tif) succeeds as GTiff. Processing input file /Users/mloskot/data/casil/remote_sensing/naip_2005/doqqs/n34119h8sw.tif. OGRCT: Source: +proj=utm +zone=11 +ellps=GRS80 +datum=NAD83 +units=m +no_defs OGRCT: Target: +proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs OGRCT: Source: +proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs OGRCT: Target: +proj=utm +zone=11 +ellps=GRS80 +datum=NAD83 +units=m +no_defs GDAL: GDALWarpKernel()::GWKNearestNoMasksByte() Src=0,0,1699x3820 Dst=0,0,1868x3602 0...10GDAL: GDALWarpKernel()::GWKNearestNoMasksByte() Src=1579,0,1813x3870 Dst=1868,0,1869x3602 ...20.GDAL: GDALWarpKernel()::GWKNearestNoMasksByte() Src=0,3768,1580x3972 Dst=0,3602,1868x3603 ..30..GDAL: GDALWarpKernel()::GWKNearestNoMasksByte() Src=1461,3819,1813x3921 Dst=1868,3602,1869x3603 .40...50GDAL: GDALWarpKernel()::GWKNearestNoMasksByte() Src=3273,0,1811x3920 Dst=3737,0,1868x3602 ...60GDAL: GDALWarpKernel()::GWKNearestNoMasksByte() Src=4966,0,1584x3970 Dst=5605,0,1869x3602 ...70.GDAL: GDALWarpKernel()::GWKNearestNoMasksByte() Src=3156,3869,1811x3871 Dst=3737,3602,1868x3603 ..80..GDAL: GDALWarpKernel()::GWKNearestNoMasksByte() Src=4849,3919,1701x3821 Dst=5605,3602,1869x3603 .90...100 - done. GDAL: GDALClose(/Users/mloskot/data/casil/remote_sensing/naip_2005/doqqs/n34119h8sw.tif) GDAL: GDALClose(out.tiff) GDAL: GDALDeregister_GTiff() called.
- gdalinfo out.tiff
~/dev/gdal/bugs/2033 $ gdalinfo out.tiff GDAL: GDALOpen(out.tiff) succeeds as GTiff. Driver: GTiff/GeoTIFF Files: out.tiff Size is 7474, 7205 Coordinate System is: GEOGCS["WGS 84", DATUM["WGS_1984", SPHEROID["WGS 84",6378137,298.2572235630016, AUTHORITY["EPSG","7030"]], AUTHORITY["EPSG","6326"]], PRIMEM["Greenwich",0], UNIT["degree",0.0174532925199433], AUTHORITY["EPSG","4326"]] Origin = (-120.005830387315683,34.941970857729622) Pixel Size = (0.000009916509033,-0.000009916509033) Metadata: AREA_OR_POINT=Area Image Structure Metadata: COMPRESSION=JPEG INTERLEAVE=PIXEL OGRCT: Source: +proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs OGRCT: Target: +proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs Corner Coordinates: Upper Left (-120.0058304, 34.9419709) (120d 0'20.99"W, 34d56'31.10"N) Lower Left (-120.0058304, 34.8705224) (120d 0'20.99"W, 34d52'13.88"N) Upper Right (-119.9317144, 34.9419709) (119d55'54.17"W, 34d56'31.10"N) Lower Right (-119.9317144, 34.8705224) (119d55'54.17"W, 34d52'13.88"N) Center (-119.9687724, 34.9062466) (119d58'7.58"W, 34d54'22.49"N) Band 1 Block=512x512 Type=Byte, ColorInterp=Red Band 2 Block=512x512 Type=Byte, ColorInterp=Green Band 3 Block=512x512 Type=Byte, ColorInterp=Blue GDAL: GDALClose(out.tiff) GDAL: GDALDeregister_GTiff() called.
- gdaladdo out.tiff 2 4 8 16
~/dev/gdal/bugs/2033 $ gdaladdo out.tiff 2 4 8 16 GDAL: GDALOpen(out.tiff) succeeds as GTiff. 0...10...20...30.tif_jpeg.c:943: failed assertion `sp->cinfo.comm.is_decompressor' Abort trap
- gdalinfo out.tiff
~/dev/gdal/bugs/2033 $ gdalinfo out.tiff GTiff: Opened 3737x3603 overview.
GTiff: Opened 1869x1802 overview.
GTiff: Opened 935x901 overview.
GTiff: Opened 468x451 overview.
GDAL: GDALOpen(out.tiff) succeeds as GTiff. Driver: GTiff/GeoTIFF Files: out.tiff Size is 7474, 7205 Coordinate System is: GEOGCS["WGS 84",
DATUM["WGS_1984",
SPHEROID["WGS 84",6378137,298.2572235630016,
AUTHORITY["EPSG","7030"]],
AUTHORITY["EPSG","6326"]],
PRIMEM["Greenwich",0], UNIT["degree",0.0174532925199433], AUTHORITY["EPSG","4326"]]
Origin = (-120.005830387315683,34.941970857729622) Pixel Size = (0.000009916509033,-0.000009916509033) Metadata:
AREA_OR_POINT=Area
Image Structure Metadata:
COMPRESSION=JPEG INTERLEAVE=PIXEL
OGRCT: Source: +proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs OGRCT: Target: +proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs Corner Coordinates: Upper Left (-120.0058304, 34.9419709) (120d 0'20.99"W, 34d56'31.10"N) Lower Left (-120.0058304, 34.8705224) (120d 0'20.99"W, 34d52'13.88"N) Upper Right (-119.9317144, 34.9419709) (119d55'54.17"W, 34d56'31.10"N) Lower Right (-119.9317144, 34.8705224) (119d55'54.17"W, 34d52'13.88"N) Center (-119.9687724, 34.9062466) (119d58'7.58"W, 34d54'22.49"N) Band 1 Block=512x512 Type=Byte, ColorInterp=Red
Overviews: 3737x3603, 1869x1802, 935x901, 468x451
Band 2 Block=512x512 Type=Byte, ColorInterp=Green
Overviews: 3737x3603, 1869x1802, 935x901, 468x451
Band 3 Block=512x512 Type=Byte, ColorInterp=Blue
Overviews: 3737x3603, 1869x1802, 935x901, 468x451
GDAL: GDALClose() GDAL: GDALClose() GDAL: GDALClose() GDAL: GDALClose() GDAL: GDALClose(out.tiff) GDAL: GDALDeregister_GTiff() called. }}}
- Comparing Chris' file with mine
~/dev/gdal/bugs/2033 $ ls -l total 74288 -rw-r--r-- 1 mloskot mloskot 19016502 Nov 28 08:50 n34.tiff -rw-r--r-- 1 mloskot mloskot 19016502 Nov 28 09:48 out.tiff
Summary:
- The assertion is different and occurs in function JPEGPreDecode but not JPEGPreEncode
- Inspite of the assertion catch, overviews are burned.
by , 16 years ago
Attachment: | openev-fwtools-2.0.1-out-tif.png added |
---|
Tested output out.tif file in OpenEV (FWTools 2.0.1, GDAL 1.5.0dev)
by , 16 years ago
Attachment: | openev-fwtools-2.0.1-out-tif-2.png added |
---|
Zooming tested output out.tif file in OpenEV (FWTools 2.0.1, GDAL 1.5.0dev)
by , 16 years ago
Attachment: | openev-fwtools-1.3.9-out-tif.png added |
---|
Tested output out.tif file in OpenEV (FWTools 1.3.9, GDAL 1.5.0dev - 1 month older, from 2007/10/11)
comment:4 by , 16 years ago
One important note, the tests I run above was made using GDAL + external libjpeg.
Here is run using internal libjpeg:
$ gdaladdo out.tiff 2 4 8 16 GTiff: Opened 3737x3603 overview. GTiff: Opened 1869x1802 overview. GTiff: Opened 935x901 overview. GTiff: Opened 468x451 overview. GDAL: GDALOpen(out.tiff) succeeds as GTiff. 0...10...tif_jpeg.c:1552: failed assertion `!sp->cinfo.comm.is_decompressor' Abort trap
comment:5 by , 16 years ago
Here is a dump of my thoughts related to this problem I expressed on the #gdal:
Nov 28 14:16:32 <mloskot> there is only one place where is_decompressor is set to false Nov 28 14:16:38 <mloskot> jcapimin.c:55 Nov 28 14:17:03 <mloskot> and execution never walks that way during processing our file Nov 28 14:34:16 <mloskot> I'm 90% sure the problem is with uninitialized struct field. Nov 28 14:34:22 <mloskot> The decompression goes fine Nov 28 14:34:55 <mloskot> but when compression of tiles starts, execution never jumps into JPEGPreEncode so is_decompress is never initialized with FALSE Nov 28 14:35:10 <mloskot> but this function is the only place where initialization of this member occurs Nov 28 14:36:03 <mloskot> Unfortunately, I'm not able to understand or conclude why it does not jump into JPEGPreEncode Nov 28 14:39:43 <mloskot> I *suppose*, the engine uses the same handle to TIFF for decoding so for encoding, but for encoding it's still "configured" with decompressor Nov 28 14:39:56 <mloskot> so, it's not correct for encoding in further stage of the process Nov 28 14:40:12 <mloskot> I mean handle GTiffDataset::hTIFF and tiff tile encoder fails Nov 28 14:41:25 <FrankW> It is possible to switch between encoding and decoding but unfortunately the encoder/decoder code for libtiff was not originally intended to do this and my efforts to support it have been not entirely comprehensive. Nov 28 14:44:43 <mloskot> FrankW: But gdaladdo -> gtiff driver uses the same object, comparing memory addresses of TIFF handle Nov 28 14:45:26 <mloskot> perhaps the gtiff driver could validate state on higher level instead of deeply in libjpeg and reconfigure if possible Nov 28 14:45:52 <mloskot> but I'm a green in this subject and have no idea if these comments make sense at all
comment:6 by , 16 years ago
Cc: | added |
---|
comment:7 by , 16 years ago
Cc: | added |
---|
I'm also hitting the same bug when just translating a big image to a tiled JPEG Tiff. I've grabbed BMNG's PNG, like world.topo.bathy.200406.3x21600x21600.A2.png (http://worldwind28.arc.nasa.gov/public/500m_png) Then I do
gdal_translate -a_srs EPSG:4326 -co "COMPRESS=JPEG" -co "TILED=YES" ../../bmng/world.topo.bathy.200406.3x21600x21600.A2.png world.topo.bathy.200406.3x21600x21600.A2.tif
And I get after a few seconds:
tif_jpeg.c:945: JPEGPreDecode: Assertion `sp->cinfo.comm.is_decompressor' failed
In fact, this is more an issue with GDALDatasetCopyWholeRaster. I've recently added a check so that we don't read more than nTargetSwathSize (10 MB) data at a time. So we can get only 10 000 000 / (21 600 * 3) = 154 lines at a time. But my destination tiff is tiled, with default tile size 256 x 256. So as 154 < 256, the tiff driver must re-read already partially jpeg-encoded blocks... That's bad because it means jpeg decompression / jpeg recompression of the same block, which lowers quality. And that's bad as it hits this bug...
GDALDatasetCopyWholeRaster should be modified so that if nMemoryPerLine * nBlockYSize > nTargetSwathSize, we still grab nBlockYSize lines at a time, but not whole lines at the same time, just multiples of nBlockXSize. So that we don't have to read already encoded tiles. I hope that make sense
comment:8 by , 16 years ago
I'm attaching a patch that seems to fix this bug. I've been able to gdaladdo on crschmidt's dataset translated to a tiled JPEG GTiff.
However there is something I don't understand. The JPEG GTiff without overviews is 15 MB large. Once "gdaladdo 2 4 8 16" has been run, it is 31 MB large.
But if I make the original JPEG GTiff write only and run COMPRESS_OVERVIEW=JPEG gdaladdo 2 4 8 16, it produces an external geotiff that it is 7 MB large. 15 + 7 = 22 MB < 31 MB...
I'm observing the samething on my BMNG JPEG GTiff : 23 + 11 = 35 < 49 MB.
by , 16 years ago
Attachment: | libtiff_jpeg_bug2033.patch added |
---|
comment:9 by , 16 years ago
Please read above : "... make the original JPEG GTiff READ-ONLY ..." and 23 + 11 = 34 ;-)
comment:10 by , 16 years ago
I'm attaching another patch that rework GDALDatasetCopyWholeRaster so that we still grab nBlockYSize lines in the swath, but not on the whole dataset width, in case of large datasets.
comment:11 by , 16 years ago
Summary: | gdaladdo tif_jpeg assertion failure: sp->cinfo.comm.is_decompressor → [PATCH] gdaladdo tif_jpeg assertion failure: sp->cinfo.comm.is_decompressor |
---|
by , 16 years ago
Attachment: | gdal_svn_trunk_rework_GDALDatasetCopyWholeRaster.patch added |
---|
Handle the case of big datasets and alignment of swath dimensions on the blocks of the destination dataset
comment:12 by , 16 years ago
Resolution: | → fixed |
---|---|
Status: | new → closed |
I confirmed getting the is_decompressor error when doing a gdal_translate of a wide image to a tiled jpeg compressed tiff, and then adding overviews.
I applied Even's patch with some cleanup to libtiff CVS head (libtiff 4), and then migrated that into the internal libtiff of GDAL (r13310).
I tried applying the patch in libtiff 3.9 but gdaladdo failed differently. I was unable to determine if this was due to the patch, or unrelated problems but I abandoned that effort. So libtiff 3.9 and earlier versions are likely to continue having problems with adding overviews on jpeg compressed tiff files and some other random read/write tasks (like gdalwarp).
I'm not prepared to apply the GDALDatasetCopyWholeRaster patch at this time. I feel that it is covering over problems in particular drivers, and/or just a different approach for lossily compressed formats.
Failure looks like this: