gdaladdo & tiffs - wrong stripbytecounts warnings and block errors

[kyngchaos@…]

When trying to ad overviews to tiffs, at higher levels I get Wrong "StripByteCounts" field warnings and 
IReadBlock and GetBlockRef failed errors.  Sometimes bus errors or seg faults.  GDAL 1.3.2, libtiff 3.8.2.  
Something like:

/usr/local/gislibs/bin/gdaladdo -r average /Artemis/public/spot-veg/ndv/2006/dv06031_wrld.tif 2 4 
8 16 32 64 128
0.Warning 1: TIFFReadDirectory:/Artemis/public/spot-veg/ndv/2006/
dv06031_wrld.tif:
Wrong "StripByteCounts" field, ignoring and calculating from
imagelength Warning 1: TIFFReadDirectory:/Artemis/public/spot-veg/ndv/2006/
dv06031_wrld.tif: Wrong "StripByteCounts" field, ignoring and calculating from imagelength
..10...20...30...40...50...60...70...80...90..Warning 1:
TIFFReadDirectory:/Artemis/public/spot-veg/ndv/2006/
dv06031_wrld.tif: Wrong"StripByteCounts" field, ignoring and calculating from imagelength
Warning 1: TIFFReadDirectory:/Artemis/public/spot-veg/ndv/2006/
dv06031_wrld.tif: Wrong "StripByteCounts" field, ignoring and calculating from imagelength
ERROR 1: /Artemis/public/spot-veg/ndv/2006/dv06031_wrld.tif:DumpModeDecode:
Not enough data for scanline 0 ERROR 1: TIFFReadEncodedTile() failed.

ERROR 1: IReadBlock failed at X offset 0, Y offset 4
ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 4
ERROR 1: /Artemis/public/spot-veg/ndv/2006/dv06031_wrld.tif:DumpModeDecode:
Not enough data for scanline 0 ERROR 1: TIFFReadEncodedTile() failed.


On Mac OS X I get this trace in the crash report for a block error:

Command: gdaladdo
Path:    /usr/local/gislibs/bin/gdaladdo
Parent:  bash [15756]

Version: ??? (???)

PID:    15882
Thread: 0

Exception:  EXC_BAD_ACCESS (0x0001)
Codes:      KERN_PROTECTION_FAILURE (0x0002) at 0x00000810

Thread 0 Crashed:
0   <<00000000>> 	0xffff862c __bzero + 44 (cpu_capabilities.h:187)
1   libtiff.3.dylib 	0x0072f30c LZWPreDecode + 284 (icplusplus.c:28)
2   libtiff.3.dylib 	0x00737f38 TIFFReadEncodedTile + 140 (icplusplus.c:28)
3   libgdal.1.dylib 	0x01059f18 GTiffRasterBand::IReadBlock(int, int, void*) + 296 (icplusplus.c:28)
4   libgdal.1.dylib 	0x010f7824 GDALRasterBand::GetLockedBlockRef(int, int, int) + 256 
(jpeg2000dataset.cpp:1188)
5   libgdal.1.dylib 	0x010fbb14 GDALRasterBand::IRasterIO(GDALRWFlag, int, int, int, int, void*, int, 
int, GDALDataType, int, int) + 984 (jpeg2000dataset.cpp:1188)
6   libgdal.1.dylib 	0x010f9cb0 GDALRegenerateOverviews + 1036 (jpeg2000dataset.cpp:1188)
7   libgdal.1.dylib 	0x010f9b24 GDALRegenerateOverviews + 640 (jpeg2000dataset.cpp:1188)
8   libgdal.1.dylib 	0x0105c400 GTiffDataset::IBuildOverviews(char const*, int, int*, int, int*, int (*)
(double, char const*, void*), void*) + 1748 (icplusplus.c:28)
9   libgdal.1.dylib 	0x010ec6d8 GDALDataset::BuildOverviews(char const*, int, int*, int, int*, int (*)
(double, char const*, void*), void*) + 188 (jpeg2000dataset.cpp:1188)
10  gdaladdo        	0x00002830 main + 332 (crt.c:355)
11  gdaladdo        	0x00001f10 _start + 340 (crt.c:272)
12  gdaladdo        	0x00001db8 start + 60

using average_mp resampling I could get a seg fault (and other ways, see below), the thread looks like:

0   libtiff.3.dylib 	0x0072fd94 cl_hash + 52 (icplusplus.c:28)
1   libtiff.3.dylib 	0x0072fe78 LZWPreEncode + 152 (icplusplus.c:28)
2   libtiff.3.dylib 	0x0073a50c TIFFWriteEncodedTile + 404 (icplusplus.c:28)
3   libgdal.1.dylib 	0x01059cf8 GTiffRasterBand::IWriteBlock(int, int, void*) + 156 (icplusplus.c:28)
4   libgdal.1.dylib 	0x010f71bc GDALRasterBand::FlushBlock(int, int) + 300 (jpeg2000dataset.cpp:
1188)
5   libgdal.1.dylib 	0x010f7250 GDALRasterBand::FlushCache() + 80 (jpeg2000dataset.cpp:1188)
6   libgdal.1.dylib 	0x010fa650 GDALRegenerateOverviews + 3500 (jpeg2000dataset.cpp:1188)
...

I installed some previous versions of GDAL and libtiff for testing, and came up with this for a greyscale, 
uncompressed tiff and a smaller color LZW compressed tiff:

           compress             levels
Versions      OV     resample     OK     error
------------------------------------------
SPOT grey, none
GDAL 1.2.6/TIFF 3.7.3
           default   nearest    2..128
            (none)   average    2..128
GDAL 1.3.0/TIFF 3.7.3
           default   average    2..128
GDAL 1.3.0/TIFF 3.7.3
           default   average    2..128
GDAL 1.3.1/TIFF 3.8.0
	       default   nearest    2,4      StripByteCounts warnings always
                                          block errors at lvl 8
                     average    2,4      StripByteCounts warnings always
                                          block errors at lvl 8
GDAL 1.3.1/TIFF 3.8.1
           default   nearest    2,4      StripByteCounts warnings always
                                          block errors at lvl 8
                     average    2,4      StripByteCounts warnings always
                                          block errors at lvl 8

small color, LZW
GDAL 1.2.6/TIFF 3.7.3
           default   nearest    2..128
            (LZW)    average    2        bus error at lvl 4
GDAL 1.3.0/TIFF 3.7.3
           default   nearest    2..128
            (LZW)    average    2        bus error at lvl 4
GDAL 1.3.0/TIFF 3.7.3
           default   nearest    2..128
            (LZW)    average    2        bus error at lvl 4
GDAL 1.3.1/TIFF 3.8.0
           default   nearest    2..128
            (LZW)    average    2        bus error at lvl 4
GDAL 1.3.1/TIFF 3.8.1
           default   nearest    2..128
            (LZW)    average    2        bus error at lvl 4
                     average_mp 2        seg fault at lvl 4

There seems to be some connection with LZW compression and average (and others?) resampling for 
the bus errors, and libtiff 3.8 is when the stripbytecount warnings and block errors start.

[warmerdam]

I have tracked down the problem, and it turns out that it was a bad
interaction with the EstimateStripByteCounts() function in libtiff's 
tif_dirread.c.  This is invoked in various circumstances where the
stripbytecount (really tilebytecount in this case) field of a TIFF file
seems to be corrupt.  This causes the values to be recomputed.

There were two issues:
 1) EstimateStripByteCount() did not have logic for tiled files, and so 
    generated wrong byte counts for the overview layers. 
 2) It should never have been invoked, but was invoked in cases where the
    first tile of an overview was written and the second was not.  The sizes
    on the subequent directory read were different resulting in a bogus
    "corrupt detection".  

I fixed both issues in libtiff (tif_dirread.c 1.86) and propogated the 
current libtiff source to GDAL's internal libtiff.

[kyngchaos@…]

Ooh, that was quick!  Does this fix depend on any other changes in the current CVS?  I'd like to apply just 
this to my 3.8.2 binary release. And maybe other critical fixes.

[warmerdam]

William, 

I am not aware of dependencies between this and other changes but I can't
be sure.  The patch to tif_dirread.c looks like this:

Index: tif_dirread.c
===================================================================
RCS file: /cvs/maptools/cvsroot/libtiff/libtiff/tif_dirread.c,v
retrieving revision 1.86
retrieving revision 1.85
diff -u -r1.86 -r1.85
--- tif_dirread.c       17 Jun 2006 17:20:49 -0000      1.86
+++ tif_dirread.c       8 Jun 2006 14:27:17 -0000       1.85
@@ -1,4 +1,4 @@
-/* $Id: tif_dirread.c,v 1.86 2006/06/17 17:20:49 fwarmerdam Exp $ */
+/* $Id: tif_dirread.c,v 1.85 2006/06/08 14:27:17 dron Exp $ */
 
 /*
  * Copyright (c) 1988-1997 Sam Leffler
@@ -681,14 +681,11 @@
                } else if (td->td_planarconfig == PLANARCONFIG_CONTIG
                           && td->td_nstrips > 2
                           && td->td_compression == COMPRESSION_NONE
-                          && td->td_stripbytecount[0] != td->td_stripbytecount[1]
-                           && td->td_stripbytecount[0] != 0 
-                           && td->td_stripbytecount[1] != 0 ) {
+                          && td->td_stripbytecount[0] !=
td->td_stripbytecount[1]) {
                        /*
-                        * XXX: Some vendors fill StripByteCount array with 
-                         * absolutely wrong values (it can be equal to 
-                         * StripOffset array, for example). Catch this case 
-                         * here.
+                        * XXX: Some vendors fill StripByteCount array with
absolutely
+                        * wrong values (it can be equal to StripOffset array, for
+                        * example). Catch this case here.
                         */
                        TIFFWarningExt(tif->tif_clientdata, module,
                "%s: Wrong \"%s\" field, ignoring and calculating from imagelength",
@@ -1019,11 +1016,6 @@
                                                                > filesize)
                        td->td_stripbytecount[i] =
                            filesize - td->td_stripoffset[i];
-       } else if( isTiled(tif) ) {
-               uint32 bytespertile = TIFFTileSize(tif);
-
-               for (i = 0; i < td->td_nstrips; i++)
-                    td->td_stripbytecount[i] = bytespertile;
        } else {
                uint32 rowbytes = TIFFScanlineSize(tif);
                uint32 rowsperstrip = td->td_imagelength/td->td_stripsperimage;

[kyngchaos@…]

oops - the patch is backwards, but I get it.  Thanks.

[kyngchaos@…]

OK, after testing, this does fix the srtipbytecounts warnings.  The bus errors are still there for LZW images.  
A different problem, so I'll add a separate bug for this.

[warmerdam]

*** Bug 1206 has been marked as a duplicate of this bug. ***