GdalRasterIO failure in Mapserver drawing large geotiff

[crschmidt@…]

I'm not sure if this is the right place for this bug report, but I think it is.

When rendering a relatively large GeoTIFF via Mapserver, I am getting random 'failed to draw layer' messages. After testing on Mapserver 4.10 and 4.9, I built 4.10 from debian sources, and debugged until I got this error:

msSetError (code=1, message_fmt=0x810ca32 "GDALRasterIO() failed:
%s", routine=0x817a9b0 "IReadBlock failed at X offset 4, Y offset 14") at maperror.c:381

This error is intermittent. If I use a query string with:

BBOX=72.7974593639504,18.986558510617186,72.88809657098165,19.04887159045117&WIDTH=528&HEIGHT=36

then there is an error.

If I use:

BBOX=72.82011866570821,19.002136780575682,72.86543726922383,19.033293320492675&WIDTH=528&HEIGHT=363

There is no error.

The tarball set to the URL of this bug contains the (very simple) mapfile I was using, as well as the image (which is quite large -- 200MB) which caused the error. The tarball is approx 90MB to download.

The image being rendered has been created using gdalwarp and gdaltranslate from ​http://labs.metacarta.com/rectifier/maps/bombay1969.jpg

gdal_translate -a_srs '+init=epsg:4326' -of VRT \
   $LIBPATH/bombay1969.jpg maps/$$.vrt $GCPS

gdalwarp 0 maps/$$.vrt maps/bombay1969.tiff -dstalpha \
   -co TILED=YES -co COMPRESS=PACKBITS -co ALPHA=YES

$GCPS is:

-gcp 2620.69249001997 4565.3926521956 72.8539037704468 19.0201511019564 -gcp 2420.7133233533 4642.0313747505 72.8477239608765 19.0178587490128 -gcp 2682.96145209581 4351.0437250499 72.8551697731018 19.0262571267574 -gcp 1880.64982534931 7592.62219311376 72.8349781036377 18.9320249486597 -gcp 1498.65369261478 7514.78599051895 72.8234338760376 18.9336892862771 -gcp 1483.08645209582 7686.02563622754 72.8231763839722 18.9288586020447 -gcp 1639.95633732536 7541.1305513972 72.8278541564941 18.9330803841945 -gcp 2135.7130738523 7640.52139471057 72.8428745269775 18.9289803857165 -gcp 1772.87662175649 7772.24419910179 72.8320169448853 18.9262605292328

This is using gdal 1.3.2 from debian unstable.

If there is any other feedback I can offer, please let me know.

[warmerdam]

The TIFF file is corrupt.  The problem seems to be in libtiff and related
to mixed read/write access for tiff files.  Without compression the gdalwarp
works fine.

... digging deeper...

[warmerdam]

The problem also goes away with sufficiently large -wm (warp memory).  If the
whole image is done in one pass (ie. -wm 900) then there is no problem. 

The problem appears to primarily affect the tiles in overlapping areas 
between "chunks" in the warper.  That is those that are read and written
multiple times, potentially changing in size. 

I'm not up to resolving this tonight, but I think it can likely be 
reproduce with a small file with several tiles where tiles are
read and re-written with increasing information content so they need
to move to the end of file. 

...

[warmerdam]

I have corrected a bug in TIFFAppendToStrip() in libtiff - resetting the
sorted flag when a strip or tile moves.  This corrects the problem though
this area of code still seems very suspect to me.