gdalwarp to compressed TIFF

[warmerdam]

The following command results in a crash with the following traceback.

gdalwarp -ts 33000 33000 openev/utm.tif out.tif -rcs -co COMPRESS=DEFLATE -ot Int16

(gdb) where
#0  0x008f32e7 in inflate_blocks_reset ()
    at /usr/include/c++/3.3.3/bits/stl_tree.h:327
#1  0x008f2618 in inflateReset () at /usr/include/c++/3.3.3/bits/stl_tree.h:327
#2  0x0041a8f8 in ZIPPreDecode (tif=0x92a1d88, s=0) at tif_zip.c:114
#3  0x004181cd in TIFFStartStrip (tif=0x92a1d88, strip=0) at tif_read.c:565
#4  0x004177dd in TIFFReadEncodedStrip (tif=0x92a6738, strip=0, buf=0xd381738, 
    size=66000) at tif_read.c:158
#5  0x0063355b in GTiffRasterBand::IReadBlock (this=0x92a6fc8, nBlockXOff=0, 
    nBlockYOff=0, pImage=0xd381738) at geotiff.cpp:483
#6  0x007567df in GDALRasterBand::GetBlockRef (this=0x92a6fc8, nXBlockOff=0, 
    nYBlockOff=0, bJustInitialize=0) at gdalrasterband.cpp:954
#7  0x00759e91 in GDALRasterBand::IRasterIO (this=0x92a6fc8, eRWFlag=GF_Write, 
    nXOff=4125, nYOff=0, nXSize=4125, nYSize=4125, pData=0xf2e83008, 
    nBufXSize=4125, nBufYSize=4125, eBufType=GDT_Int16, nPixelSpace=2, 
    nLineSpace=8250) at rasterio.cpp:172
#8  0x0074ddeb in GDALDataset::IRasterIO (this=0x92a1dd8, eRWFlag=GF_Write, 
    nXOff=4125, nYOff=0, nXSize=4125, nYSize=4125, pData=0xf2e83008, 
    nBufXSize=4125, nBufYSize=4125, eBufType=GDT_Int16, nBandCount=1, 
    panBandMap=0x92a65e0, nPixelSpace=2, nLineSpace=8250, nBandSpace=34031250)
    at gdaldataset.cpp:1278
#9  0x0074e0b1 in GDALDataset::RasterIO (this=0x92a1dd8, eRWFlag=GF_Write, 
    nXOff=4125, nYOff=0, nXSize=4125, nYSize=4125, pData=0xf2e83008, 
    nBufXSize=4125, nBufYSize=4125, eBufType=GDT_Int16, nBandCount=1, 
    panBandMap=0x92a65e0, nPixelSpace=2, nLineSpace=8250, nBandSpace=34031250)
    at gdaldataset.cpp:1455
#10 0x0074e124 in GDALDatasetRasterIO (hDS=0x92a1dd8, eRWFlag=GF_Write, 
    nXOff=4125, nYOff=0, nXSize=4125, nYSize=4125, pData=0xf2e83008, 
    nBufXSize=4125, nBufYSize=4125, eBufType=GDT_Int16, nBandCount=1, 
    panBandMap=0x92a65e0, nPixelSpace=0, nLineSpace=0, nBandSpace=0)
    at gdaldataset.cpp:1489
#11 0x0077bab0 in GDALWarpOperation::WarpRegion (this=0xfee63070, 
    nDstXOff=4125, nDstYOff=0, nDstXSize=4125, nDstYSize=4125, nSrcXOff=64, 
    nSrcYOff=0, nSrcXSize=65, nSrcYSize=65) at gdalwarpoperation.cpp:1190
#12 0x0077ada5 in GDALWarpOperation::ChunkAndWarpImage (this=0xfee63070, 
    nDstXOff=0, nDstYOff=0, nDstXSize=33000, nDstYSize=33000)
    at gdalwarpoperation.cpp:683
#13 0x0804aeb1 in main (argc=11, argv=0x92a24b8) at gdalwarp.cpp:657

Presumably this is related to rewriting a compressed file.

[warmerdam]

I get a failure if writing to a PACKBITS compressed output file as well, though
not a crash. 

The utm.tif is:

  ftp://ftp.remotesensing.org/gdal/data/gtiff/utm.tif

[warmerdam]

*** Bug 790 has been marked as a duplicate of this bug. ***

[warmerdam]

It appears #800 is related or even a duplicate of this bug. 

[warmerdam]

This problem still exists.  Charles at ERMapper reports:

As near as I can tell, this is what is happening (I'm afraid my knowledge of
buffered I/O is limited, so I hope this makes some sort of sense):

- At some point while GDAL is reading from the input file and writing to the
output file/cache, the cache limit is hit and GDAL decides to flush the
block cache.
- In order to do that, it needs to encode the output data, and thus needs to
set up the encoder. The call sequence looks like this
	PredictorSetupEncode - line 168 [tif_predict.c]
	TIFFWriteEncodedStrip - line 220 [tif_write.c]
	GTiffDataset::FlushBlockBuf - line 1717 [gtiff\geotiff.cpp]
	GTiffDataset::LoadBlockBuf - line 1760 [gtiff\geotiff.cpp]
	GTiffRasterBand::IWriteBlock - line 702 [gtiff\geotiff.cpp]
	GDALRasterBlock::Write - line 376 [gdalrasterblock.cpp]
	GDALRasterBand::FlushBlock - line 990 [gdalrasterband.cpp]
	GDALRasterBlock::FlushCacheBlock - line 242 [gdalrasterblock.cpp]
	GDALFlushCacheBlock - line 205 [gdalrasterblock.cpp]
	GDALRasterBlock::Internalize - line 458 [gdalrasterblock.cpp]
	etc.
- After calling PredictorSetupEncode, TIFFWriteStrip sets a flag to indicate
the encode/decoder is set up
	tif->tif_flags |= TIFF_CODERSETUP;
  (It's possible that this would also happen while closing the output file
if the flag wasn't already set)
- Everything is fine until the output file is closed. In the process of
deleting the dataset, the output cache is flushed. For some reason, this
requires a call to TIFFReadEncodedStrip. I haven't been able to puzzle
through the sequence of encoding/decoding, so I'm not exactly sure why.
- This results in a call to TIFFStartStrip, which knows that some decoding
may be necessary, and checks whether the decoder has been setup. If
necessary, the decoder is initialised.
		if ((tif->tif_flags & TIFF_CODERSETUP) == 0) {
		if (!(*tif->tif_setupdecode)(tif))

Now, the decoder is _not_ set up, but the flag in question _is_ set, due to
the earlier encoder initialisation.
This is a problem, because LZWPreDecode initialises what I assume is some
sort of decoder table
	sp->dec_free_entp = sp->dec_codetab + CODE_FIRST;
	/*
	 * Zero entries that are not yet filled in.  We do
	 * this to guard against bogus input data that causes
	 * us to index into undefined entries.  If you can
	 * come up with a way to safely bounds-check input codes
	 * while decoding then you can remove this operation.
	 */
	_TIFFmemset(sp->dec_free_entp, 0, (CSIZE-CODE_FIRST)*sizeof (code_t));
Unfortunately, sp->dec_codetab is NULL since it is allocated in
LZWSetupDecode decode, which has never been called. As a result, the memset
fails badly.

So essentially, the problem seems to be that the flag corresponding to
TIFF_CODERSETUP is used to indicate that both the encoder and decoder have
been initialised, when in reality only one of them may have been.

It is possible we are missing a step that results in that flag being
cleared, but I haven't been able to see anything. Any suggestions?

Bumping this in priority, working on today.

[warmerdam]

BTW, the following (with a default GDAL_CACHEMAX setting) reproduces
the problem much faster.  The key to reproducing it with gdalwarp 
seems to be a small warp memory cache. 

  gdalwarp -ts 6000 6000 openev/utm.tif out.tif -co COMPRESS=DEFLATE -wm 1

[warmerdam]

I have made changes in the libtiff files tif_lzw.c and tif_zip.c which
seems to fix this problem properly. I have committed these in libtiff, and
also updated the GDAL version of these files. 
I have also added tests for lzw, zip and packbits compression in the 
autotest/gcore/tiff_write.py script. 

Note, that tif_jpeg still seems to suffer from this state problem.