Ticket #2005 (closed defect: fixed)

Opened 6 months ago

Last modified 1 month ago

Creation of large TIFF fails with some block sizes

Reported by: rouault Assigned to: warmerdam
Priority: high Milestone: 1.5.1
Component: GDAL_Raster Version: svn-trunk
Severity: normal Keywords: geotiff big dataset gtiff libtiff
Cc: rouault, mloskot, wollez@gmx.net

Description

gdal_translate to GTiff on the attached WMS dataset fails or not depending on the value of the block size or tile/strip mode (tests done on GDAL svn trunk, with libtiff internal)

1) CPL_DEBUG=GDAL apps/gdal_translate test_wms.txt tt.tif fails very quickly : 

GDAL: GDALOpen(test_wms.txt) succeeds as WMS.
Input file size is 2666666, 1333333
0ERROR 1: TIFFFetchStripThing:Sanity check on size of "StripOffsets" value failed
ERROR 1: TIFFFetchStripThing:Sanity check on size of "StripOffsets" value failed
GDAL: GDALClose(test_wms.txt)
GDAL: GDALDeregister_GTiff() called.

2) CPL_DEBUG=GDAL apps/gdal_translate -co "TILED=YES" test_wms.txt tt.tif takes huge amount of RAM (disk swapping) but doesn't fail. When interrupted by Ctrl+C, tt.tif is of size 0. 

GDAL: GDALOpen(test_wms.txt) succeeds as WMS.
Input file size is 2666666, 1333333
0

3) CPL_DEBUG=GDAL apps/gdal_translate -co "TILED=YES" -co "BLOCKYSIZE=1024" -co "BLOCKXSIZE=1024" test_wms.txt tt.tif fails very quickly : 

GDAL: GDALOpen(test_wms.txt) succeeds as WMS.
Input file size is 2666666, 1333333
0ERROR 1: TIFFFetchStripThing:Sanity check on size of "TileOffsets" value failed
ERROR 1: TIFFFetchStripThing:Sanity check on size of "TileOffsets" value failed
GDAL: GDALClose(test_wms.txt)
GDAL: GDALDeregister_GTiff() called.

4) CPL_DEBUG=GDAL apps/gdal_translate -co "TILED=YES" -co "BLOCKYSIZE=1024" -co "BLOCKXSIZE=16384" test_wms.txt tt.tif begins to download data from the WMS server 

GDAL: GDALOpen(test_wms.txt) succeeds as WMS.
Input file size is 2666666, 1333333
0GDAL: GDALOpen(tt.tif) succeeds as GTiff.
GDAL: GDALDatasetCopyWholeRaster(): adjusting to 1 line swath since requirement (1024 * 7999998 bytes) exceed target swath size (10000000 bytes)
GDAL: GDALDatasetCopyWholeRaster(): 1 line swath, bInterleave=1

If i interrupt with Ctrl+C and gdalinfo tt.tif, I get the following expected result : 

Driver: GTiff/GeoTIFF
Files: tt.tif
Size is 2666666, 1333333
Coordinate System is:
GEOGCS["WGS 84",
    DATUM["WGS_1984",
        SPHEROID["WGS 84",6378137,298.2572235629972,
            AUTHORITY["EPSG","7030"]],
        AUTHORITY["EPSG","6326"]],
    PRIMEM["Greenwich",0],
    UNIT["degree",0.0174532925199433],
    AUTHORITY["EPSG","4326"]]
Origin = (-180.000000000000000,90.000000000000000)
Pixel Size = (0.000135000033750,-0.000135000033750)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  (-180.0000000,  90.0000000) (180d 0'0.00"W, 90d 0'0.00"N)
Lower Left  (-180.0000000, -90.0000000) (180d 0'0.00"W, 90d 0'0.00"S)
Upper Right ( 180.0000000,  90.0000000) (180d 0'0.00"E, 90d 0'0.00"N)
Lower Right ( 180.0000000, -90.0000000) (180d 0'0.00"E, 90d 0'0.00"S)
Center      (  -0.0000000,   0.0000000) (  0d 0'0.00"W,  0d 0'0.00"N)
Band 1 Block=16384x1024 Type=Byte, ColorInterp=Red
Band 2 Block=16384x1024 Type=Byte, ColorInterp=Green
Band 3 Block=16384x1024 Type=Byte, ColorInterp=Blue

Attachments

test_wms.txt (0.6 kB) - added by rouault on 11/19/07 15:17:08.

Change History

11/19/07 15:17:08 changed by rouault

  • attachment test_wms.txt added.

11/21/07 14:12:05 changed by warmerdam

  • keywords changed from geotiff big dataset to geotiff big dataset gtiff libtiff.
  • status changed from new to assigned.

It appears the following test is to zealous, assuming any tag contents larger than 4MB must indicate corruption. I have emailed Joris via the tiff list about the issue. 

static enum TIFFReadDirEntryErr TIFFReadDirEntryArray(TIFF* tif, TIFFDirEntry* direntry, uint32* count, uint32 desttypesize, void** value)
{
	int typesize;
	uint32 datasize;
	void* data;
	typesize=TIFFDataWidth(direntry->tdir_type);
	if ((direntry->tdir_count==0)||(typesize==0))
	{
		*value=0;
		return(TIFFReadDirEntryErrOk);
	}
	if ((uint64)(4*1024*1024/typesize)<direntry->tdir_count)
		return(TIFFReadDirEntryErrSizesan);
	if ((uint64)(4*1024*1024/desttypesize)<direntry->tdir_count)
		return(TIFFReadDirEntryErrSizesan);

03/07/08 14:12:11 changed by warmerdam

  • priority changed from normal to high.
  • cc set to rouault, mloskot.
  • milestone set to 1.5.1.

Elevating priority and moving to 1.5.1 milestone. I think we need to get this fix out in the libtiff version embedded in GDAL for 1.5.1. I'll take care of it.

03/07/08 14:15:35 changed by rouault

  • cc changed from rouault, mloskot to rouault, mloskot, wollez@gmx.net.

03/14/08 02:02:23 changed by warmerdam

I have #ifdef'ed out the tag size sanity checks in libtiff4 (CVS head). The updated tif_dirread.c has been downstreamed into trunk (r14002) and 1.5 branch (r14003).

Any validiation of the fixes would be appreciated.

03/14/08 09:50:40 changed by warmerdam

I have pulled down a full refresh of libtiff in trunk (r14006) since the tif_dirread.c change did not capture related needed changes for other files. In 1.5 branch I backed out the change from r14003 and applied a more narrow fix for just the tag size sanity checking (r14007).

03/14/08 18:55:54 changed by warmerdam

I botched r14007 and just backed everything out - I neglected to apply the narrow fix. I have applied this now in 1.5 branch as r14001.

04/06/08 07:05:09 changed by rouault

  • status changed from assigned to closed.
  • resolution set to fixed.

This is closed