Creation of large TIFF fails with some block sizes

[rouault]

gdal_translate to GTiff on the attached WMS dataset fails or not depending on the value of the block size or tile/strip mode (tests done on GDAL svn trunk, with libtiff internal)

1) CPL_DEBUG=GDAL apps/gdal_translate test_wms.txt tt.tif fails very quickly :

GDAL: GDALOpen(test_wms.txt) succeeds as WMS.
Input file size is 2666666, 1333333
0ERROR 1: TIFFFetchStripThing:Sanity check on size of "StripOffsets" value failed
ERROR 1: TIFFFetchStripThing:Sanity check on size of "StripOffsets" value failed
GDAL: GDALClose(test_wms.txt)
GDAL: GDALDeregister_GTiff() called.

2) CPL_DEBUG=GDAL apps/gdal_translate -co "TILED=YES" test_wms.txt tt.tif takes huge amount of RAM (disk swapping) but doesn't fail. When interrupted by Ctrl+C, tt.tif is of size 0.

GDAL: GDALOpen(test_wms.txt) succeeds as WMS.
Input file size is 2666666, 1333333
0

3) CPL_DEBUG=GDAL apps/gdal_translate -co "TILED=YES" -co "BLOCKYSIZE=1024" -co "BLOCKXSIZE=1024" test_wms.txt tt.tif fails very quickly :

GDAL: GDALOpen(test_wms.txt) succeeds as WMS.
Input file size is 2666666, 1333333
0ERROR 1: TIFFFetchStripThing:Sanity check on size of "TileOffsets" value failed
ERROR 1: TIFFFetchStripThing:Sanity check on size of "TileOffsets" value failed
GDAL: GDALClose(test_wms.txt)
GDAL: GDALDeregister_GTiff() called.

4) CPL_DEBUG=GDAL apps/gdal_translate -co "TILED=YES" -co "BLOCKYSIZE=1024" -co "BLOCKXSIZE=16384" test_wms.txt tt.tif begins to download data from the WMS server

GDAL: GDALOpen(test_wms.txt) succeeds as WMS.
Input file size is 2666666, 1333333
0GDAL: GDALOpen(tt.tif) succeeds as GTiff.
GDAL: GDALDatasetCopyWholeRaster(): adjusting to 1 line swath since requirement (1024 * 7999998 bytes) exceed target swath size (10000000 bytes)
GDAL: GDALDatasetCopyWholeRaster(): 1 line swath, bInterleave=1

If i interrupt with Ctrl+C and gdalinfo tt.tif, I get the following expected result :

Driver: GTiff/GeoTIFF
Files: tt.tif
Size is 2666666, 1333333
Coordinate System is:
GEOGCS["WGS 84",
    DATUM["WGS_1984",
        SPHEROID["WGS 84",6378137,298.2572235629972,
            AUTHORITY["EPSG","7030"]],
        AUTHORITY["EPSG","6326"]],
    PRIMEM["Greenwich",0],
    UNIT["degree",0.0174532925199433],
    AUTHORITY["EPSG","4326"]]
Origin = (-180.000000000000000,90.000000000000000)
Pixel Size = (0.000135000033750,-0.000135000033750)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  (-180.0000000,  90.0000000) (180d 0'0.00"W, 90d 0'0.00"N)
Lower Left  (-180.0000000, -90.0000000) (180d 0'0.00"W, 90d 0'0.00"S)
Upper Right ( 180.0000000,  90.0000000) (180d 0'0.00"E, 90d 0'0.00"N)
Lower Right ( 180.0000000, -90.0000000) (180d 0'0.00"E, 90d 0'0.00"S)
Center      (  -0.0000000,   0.0000000) (  0d 0'0.00"W,  0d 0'0.00"N)
Band 1 Block=16384x1024 Type=Byte, ColorInterp=Red
Band 2 Block=16384x1024 Type=Byte, ColorInterp=Green
Band 3 Block=16384x1024 Type=Byte, ColorInterp=Blue

[warmerdam]

It appears the following test is to zealous, assuming any tag contents larger than 4MB must indicate corruption. I have emailed Joris via the tiff list about the issue.

static enum TIFFReadDirEntryErr TIFFReadDirEntryArray(TIFF* tif, TIFFDirEntry* direntry, uint32* count, uint32 desttypesize, void** value)
{
	int typesize;
	uint32 datasize;
	void* data;
	typesize=TIFFDataWidth(direntry->tdir_type);
	if ((direntry->tdir_count==0)||(typesize==0))
	{
		*value=0;
		return(TIFFReadDirEntryErrOk);
	}
	if ((uint64)(4*1024*1024/typesize)<direntry->tdir_count)
		return(TIFFReadDirEntryErrSizesan);
	if ((uint64)(4*1024*1024/desttypesize)<direntry->tdir_count)
		return(TIFFReadDirEntryErrSizesan);

[warmerdam]

Elevating priority and moving to 1.5.1 milestone. I think we need to get this fix out in the libtiff version embedded in GDAL for 1.5.1. I'll take care of it.

[warmerdam]

I have #ifdef'ed out the tag size sanity checks in libtiff4 (CVS head). The updated tif_dirread.c has been downstreamed into trunk (r14002) and 1.5 branch (r14003).

Any validiation of the fixes would be appreciated.

[warmerdam]

I have pulled down a full refresh of libtiff in trunk (r14006) since the tif_dirread.c change did not capture related needed changes for other files. In 1.5 branch I backed out the change from r14003 and applied a more narrow fix for just the tag size sanity checking (r14007).

[warmerdam]

I botched r14007 and just backed everything out - I neglected to apply the narrow fix. I have applied this now in 1.5 branch as r14001.

[rouault]

This is closed