gdal_translate fails to read some specific tiff files (data missing ?)

[mike.dunne@…]

Sample tif file provided (separately to Frank) fails in gdal_translate (-of 
GTiff) with following error: 
C:\Development\gdal-1.3.1\apps>gdal_translate -of GTiff 18104755.tif junk.tif
Input file size is 3334, 1667
0...10...20...30.ERROR 1: 18104755.tif:DumpModeDecode: Not enough data for 
scanline 0
ERROR 1: TIFFReadEncodedTile() failed.
ERROR 1: IReadBlock failed at X offset 0, Y offset 13
ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 13

Have tried the file in Erdas ViewFinder and it also fails to read there.
Geomatica viewer handles it ok.
Symptom in ER Mapper (using gdal driver) is the file is read, but frequently 
displays garbage in the bottom strip of the file. (e.g. every couple of 
zoom/pans)

This file displays ok in previous versions of ER Mapper (not using gdal),
and ER Viewer.

Most other tif files are fine, but there are 4-5 files like this which have 
the same problem. We have only tested it against 1.3.1 Will test it against 
1.3.2 and daily cut and advise if issue is resolved. 

Version we are running against is 1.3.1 Will check it against 1.3.2.

[warmerdam]

Problem confirmed.  The file has some short tiles.  I'm not exactly sure 
what to do differently since the file is clearly in error. 

I have changed geotiff.cpp so that IReadBlock() memset's the imagery to 
zeros on failure. 

I will look into having gdal_translate and library functions able to continue
on when block reads fail. 

[warmerdam]

This is also related to the problem reported in:

​http://trac.osgeo.org/mapguide/ticket/268

[warmerdam]

I have added test files to the test suite to demonstrate this problem with *tiled* files. I have been unable to reproduce it with stripped files so far, though I have seen it in that context in the past (r12314, r12315).

I have committed a fix for this problem in trunk (r12316) that restricts the data requested for partial bottom strip/tiles.

[Even Rouault]

Since the r12316 commit, tiff_write_4 autotest blows up on my machine :

Here's the output under Valgrind and with some traces added at line 1861 of geotiff.cpp

GTIFF:  (int)((nBlockYOff+1) * nBlockYSize) = 64, nRasterYSize = 50,
nBlockYOff = 1, nBlockYSize = 32, nBlockBufSize = 3072, nBlockReqSize = 1728

GTIFF:  (int)((nBlockYOff+1) * nBlockYSize) = 96, nRasterYSize = 50,
nBlockYOff = 2, nBlockYSize = 32, nBlockBufSize = 3072, nBlockReqSize = -1344

GTIFF:  (int)((nBlockYOff+1) * nBlockYSize) = 128, nRasterYSize = 50,
nBlockYOff = 3, nBlockYSize = 32, nBlockBufSize = 3072, nBlockReqSize = 384

GTIFF:  (int)((nBlockYOff+1) * nBlockYSize) = 64, nRasterYSize = 50,
nBlockYOff = 1, nBlockYSize = 32, nBlockBufSize = 3072, nBlockReqSize = 1728

GTIFF:  (int)((nBlockYOff+1) * nBlockYSize) = 96, nRasterYSize = 50,
nBlockYOff = 2, nBlockYSize = 32, nBlockBufSize = 3072, nBlockReqSize = -1344
==17807==
==17807== Invalid read of size 1
==17807==    at 0x474C540: _TIFFmemcpy (tif_vsi.c:180)
==17807==    by 0x471A761: DumpModeDecode (tif_dumpmode.c:103)
==17807==    by 0x4748AB8: TIFFReadEncodedTile (tif_read.c:449)
==17807==    by 0x461A457: GTiffDataset::LoadBlockBuf(int) (geotiff.cpp:1882)
==17807==    by 0x461A98A: GTiffRasterBand::IReadBlock(int, int, void*) (geotiff.cpp:465)
==17807==    by 0x479308E: GDALRasterBand::GetLockedBlockRef(int, int, int) (gdalrasterband.cpp:1099)
==17807==    by 0x4798774: GDALRasterBand::IRasterIO(GDALRWFlag, int, int, int, int, void*, int, int, GDALDataType, int, int) (rasterio.cpp:210)
==17807==    by 0x4792751: GDALRasterBand::RasterIO(GDALRWFlag, int, int, int, int, void*, int, int, GDALDataType, int, int) (gdalrasterband.cpp:237)
==17807==    by 0x4792815: GDALRasterIO (gdalrasterband.cpp:262)
==17807==    by 0x47BB770: GDALChecksumImage (gdalchecksum.cpp:83)
==17807==    by 0x451924C: _wrap_Band_Checksum (gdal_wrap.cpp:3361)
==17807==    by 0x805C786: PyObject_Call (in /usr/bin/python2.5)
==17807==  Address 0x53867E8 is 0 bytes after a block of size 3,072 alloc'd
==17807==    at 0x4021620: malloc (vg_replace_malloc.c:149)
==17807==    by 0x47B3AD4: VSIMalloc (cpl_vsisimple.cpp:280)
==17807==    by 0x474C4A8: _TIFFmalloc (tif_vsi.c:156)
==17807==    by 0x4749434: TIFFReadBufferSetup (tif_read.c:690)
==17807==    by 0x474927F: TIFFFillTile (tif_read.c:648)
==17807==    by 0x4748A79: TIFFReadEncodedTile (tif_read.c:449)
==17807==    by 0x461A457: GTiffDataset::LoadBlockBuf(int) (geotiff.cpp:1882)
==17807==    by 0x461A98A: GTiffRasterBand::IReadBlock(int, int, void*) (geotiff.cpp:465)
==17807==    by 0x479308E: GDALRasterBand::GetLockedBlockRef(int, int, int) (gdalrasterband.cpp:1099)
==17807==    by 0x4798774: GDALRasterBand::IRasterIO(GDALRWFlag, int, int, int, int, void*, int, int, GDALDataType, int, int) (rasterio.cpp:210)
==17807==    by 0x4792751: GDALRasterBand::RasterIO(GDALRWFlag, int, int, int, int, void*, int, int, GDALDataType, int, int) (gdalrasterband.cpp:237)
==17807==    by 0x4792815: GDALRasterIO (gdalrasterband.cpp:262)
==17807==
==17807== Process terminating with default action of signal 11 (SIGSEGV)
==17807==  Bad permissions for mapped region at address 0x5754000
==17807==    at 0x474C540: _TIFFmemcpy (tif_vsi.c:180)
==17807==    by 0x471A761: DumpModeDecode (tif_dumpmode.c:103)
==17807==    by 0x4748AB8: TIFFReadEncodedTile (tif_read.c:449)
==17807==    by 0x461A457: GTiffDataset::LoadBlockBuf(int) (geotiff.cpp:1882)
==17807==    by 0x461A98A: GTiffRasterBand::IReadBlock(int, int, void*) (geotiff.cpp:465)
==17807==    by 0x479308E: GDALRasterBand::GetLockedBlockRef(int, int, int) (gdalrasterband.cpp:1099)
==17807==    by 0x4798774: GDALRasterBand::IRasterIO(GDALRWFlag, int, int, int, int, void*, int, int, GDALDataType, int, int) (rasterio.cpp:210)
==17807==    by 0x4792751: GDALRasterBand::RasterIO(GDALRWFlag, int, int, int, int, void*, int, int, GDALDataType, int, int) (gdalrasterband.cpp:237)
==17807==    by 0x4792815: GDALRasterIO (gdalrasterband.cpp:262)
==17807==    by 0x47BB770: GDALChecksumImage (gdalchecksum.cpp:83)
==17807==    by 0x451924C: _wrap_Band_Checksum (gdal_wrap.cpp:3361)
==17807==    by 0x805C786: PyObject_Call (in /usr/bin/python2.5)

[Even Rouault]

Problem with tiff_write_4 fixed in r12326

[warmerdam]

Patch back ported into the 1.4 branch (r12338).