Ticket #2486 (closed defect: worksforme)

Opened 2 years ago

Last modified 1 year ago

IReadBlock failure in gdaladdo

Reported by: antonio Assigned to: chaitanya
Priority: low Milestone: 1.6.1
Component: GDAL_Raster Version: 1.5.2
Severity: normal Keywords: GTiff
Cc: warmerdam

Description

I'm using GDAL 1.5.2 on debian sid. Trying the following: 

$ gdal_translate -of VRT HDF5:test9001.h5://S01/SBI test9001.h5.vrt
$ gdaladdo -r average test9001.h5.vrt 2 4

I get an IReadBlock failure: 

0...10...20...30...40...50...60...70...80...90..
Warning 1: TIFFReadDirectory:test9001.h5.vrt.ovr: Wrong "StripByteCounts" field, ignoring and calculating from imagelength
Warning 1: TIFFReadDirectory:test9001.h5.vrt.ovr: Wrong "StripByteCounts" field, ignoring and calculating from imagelength
ERROR 1: test9001.h5.vrt.ovr:DumpModeDecode: Not enough data for scanline 0
ERROR 1: TIFFReadEncodedTile() failed.

ERROR 1: IReadBlock failed at X offset 0, Y offset 15
ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 15
ERROR 1: test9001.h5.vrt.ovr:DumpModeDecode: Not enough data for scanline 0
ERROR 1: TIFFReadEncodedTile() failed.

[CUT]

ERROR 1: IReadBlock failed at X offset 1, Y offset 16
ERROR 1: GetBlockRef failed at X block offset 1, Y block offset 16
ERROR 1: test9001.h5.vrt.ovr:DumpModeDecode: Not enough data for scanline 128
ERROR 1: TIFFReadEncodedTile() failed.

More than 1000 errors or warnings have been reported. No more will be reported from now.
100 - done.

The problem seems do be related to the "-r average" option.

Attached the test files.

Attachments

test9001.h5.bz2 (279 bytes) - added by antonio on 07/18/08 06:35:53.

Change History

07/18/08 06:35:53 changed by antonio

  • attachment test9001.h5.bz2 added.

(follow-up: ↓ 2 ) 07/18/08 10:18:48 changed by warmerdam

  • cc set to warmerdam.
  • keywords set to GTiff.
  • component changed from default to GDAL_Raster.
  • owner changed from warmerdam to mloskot.
  • priority changed from normal to high.

Mateusz,

Could you try and reproduce this?

Antonio,

Could you establish what version of libtiff is being used on debian sid? I suspect this is a problem already fixed in libtiff but that sid is using a somewhat older version of libtiff without the fix. Also, please confirm you are using the pre-packaged GDAL for Debian that uses the system libtiff, not one you built yourself that might be using an internal libtiff.

If the problem is just an old libtiff, I don't think there will be anything we can do.

(in reply to: ↑ 1 ) 07/18/08 11:11:27 changed by antonio

Replying to warmerdam:

Mateusz, Could you try and reproduce this? Antonio, Could you establish what version of libtiff is being used on debian sid? I suspect this is a problem already fixed in libtiff but that sid is using a somewhat older version of libtiff without the fix. Also, please confirm you are using the pre-packaged GDAL for Debian that uses the system libtiff, not one you built yourself that might be using an internal libtiff.

Yes, I'm using standard Debian packages:

  • libgdal1-1.5.0 1.5.2-2 (unstable) --> depends on lintiff4
  • gdal-bin 1.5.2-2 (unstable)
  • libtiff4 3.8.2-10 (unstable)

07/18/08 18:03:32 changed by mloskot

  • status changed from new to assigned.

Fellows,

Tonight, I've only access to Windows machine. I'm back to my regular desk tomorrow and will work on this issue.

08/06/08 14:59:33 changed by mloskot

Antonio, Frank,

I've reproduced this problem on Debian Sid (unstable) using GDAL and libtiff installed from packages.

I also made tests on Ubuntu 7.04 with current version of GDAL from SVN trunk configured with option --with-libtiff=internal and HDF5 support from Debian package libhdf5-serial-dev (1.6.5). Here are my results:

  • check HDF5 support 
    $ export CPL_DEBUG=ON

$ gdalinfo test9001.h5 
GDAL: GDALOpen(test9001.h5) succeeds as HDF5.
Driver: HDF5/Hierarchical Data Format Release 5
Files: test9001.h5
Size is 512, 512
Coordinate System is `'
Subdatasets:
  SUBDATASET_0_NAME=HDF5:"test9001.h5"://S01/SBI
  SUBDATASET_0_DESC=[9001x9001] //S01/SBI (8-bit unsigned integer)
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0,  512.0)
Upper Right (  512.0,    0.0)
Lower Right (  512.0,  512.0)
Center      (  256.0,  256.0)
GDAL: GDALClose(test9001.h5)
GDAL: GDALDeregister_GTiff() called.
  • translate to VRT 
    $ gdal_translate -of VRT HDF5:test9001.h5://S01/SBI test9001.h5.vrt
GDAL: GDALOpen(HDF5:test9001.h5://S01/SBI) succeeds as HDF5Image.
Input file size is 9001, 9001
GDAL: GDALClose(test9001.h5.vrt)
GDAL: GDALClose(HDF5:test9001.h5://S01/SBI)
GDAL: GDALDeregister_GTiff() called.
  • generate overviews with gdaladdo. No IReadBlock failures occurred, only warning about potential trashing. 
    $ gdaladdo -r average test9001.h5.vrt 2 4
GDAL: GDALOpen(test9001.h5.vrt) succeeds as VRT.
GTiff: Opened 2251x2251 overview.

GDAL: GDALOpen(test9001.h5.vrt.ovr) succeeds as GTiff.
0GDAL: GDALOpen(HDF5:test9001.h5://S01/SBI) succeeds as HDF5Image.
GTiff: SetDirectory(              8)
...10...20...30...40...50...60...70...GDAL: Potential thrashing on band 1 of test9001.h5.vrt.ovr.
GTiff: SetDirectory(          10538)
80GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
.GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
.GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
.GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
90GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
.GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
.GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
.GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
100 - done.
GDAL: GDALClose()
GTiff: SetDirectory(              8)
GDAL: GDALClose(test9001.h5.vrt.ovr)
GDAL: 2592 block reads on 1296 block band 1 of test9001.h5.vrt.ovr.
GTiff: Opened 2251x2251 overview.

GDAL: GDALOpen(test9001.h5.vrt.ovr) succeeds as GTiff.
GDAL: GDALClose(test9001.h5.vrt)
GDAL: GDALClose(HDF5:test9001.h5://S01/SBI)
GDAL: GDALClose(HDF5:test9001.h5://S01/SBI)
GDAL: GDALClose()
GTiff: SetDirectory(              8)
GDAL: GDALClose(test9001.h5.vrt.ovr)
GDAL: GDALDeregister_GTiff() called.
  • inspecting generated overview 
    $ gdalinfo test9001.h5.vrt.ovr 
GTiff: Opened 2251x2251 overview.

GDAL: GDALOpen(test9001.h5.vrt.ovr) succeeds as GTiff.
Driver: GTiff/GeoTIFF
Files: test9001.h5.vrt.ovr
Size is 4501, 4501
Coordinate System is `'
Image Structure Metadata:
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0, 4501.0)
Upper Right ( 4501.0,    0.0)
Lower Right ( 4501.0, 4501.0)
Center      ( 2250.5, 2250.5)
Band 1 Block=128x128 Type=UInt16, ColorInterp=Gray
  Overviews: 2251x2251
GDAL: GDALClose()
GTiff: SetDirectory(              8)
GDAL: GDALClose(test9001.h5.vrt.ovr)
GDAL: GDALDeregister_GTiff() called.

Observing no crash of gdaladdo above, I'm inclined to support Frank's opinion the problem is in libtif version 3.8.2 used on Sid. Frank?

08/20/08 13:17:52 changed by warmerdam

  • cc changed from warmerdam to mloskot.
  • owner changed from mloskot to warmerdam.
  • status changed from assigned to new.

I'm going to try this with 1.5 branch and libtiff internal vs. libtiff 3.8.2.

08/20/08 13:28:52 changed by warmerdam

  • priority changed from high to low.
  • milestone deleted.

I was unable to briefly reproduce the problem but I didn't actually build libtiff 3.8.2 - I just used an old one that was installed on my system.

I'm dropping the milestone for this ticket since I believe it is a libtiff issue that has already been fixed.

10/26/08 11:53:21 changed by sprice

I almost opened a new ticket, but this looks like the same issue.

I'm running the latest code from GDAL trunk and TIFF stable branch, and I'm getting the same issue. In the code sample below, I first generate a fresh GeoTIFF. Then I try to add overviews in a few different manners. Note that the error is different between the first attempt and the second. The next attempts use nearest resampling, but it still fails each time.

I'll attach the file I'm using. 

seth:gdal sprice$ gdal_translate N084W180_x32_GLC.tif N084W180_x32_GLC.new.tif
Input file size is 21600, 21600
0...10...20...30...40...50...60...70...80...90...100 - done.
seth:gdal sprice$ gdaladdo -r mode N084W180_x32_GLC.new.tif 2 4
0...Warning 1: TIFFReadDirectory:N084W180_x32_GLC.new.tif: Wrong "StripByteCounts" field, ignoring and calculating from imagelength
Warning 1: TIFFReadDirectory:N084W180_x32_GLC.new.tif: Wrong "StripByteCounts" field, ignoring and calculating from imagelength
ERROR 1: N084W180_x32_GLC.new.tif:DumpModeDecode: Not enough data for scanline 0
ERROR 1: TIFFReadEncodedTile() failed.

ERROR 1: IReadBlock failed at X offset 0, Y offset 4
ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 4
10...20...30...40...50...60...70...80...90...100 - done.
Overview building failed.
seth:gdal sprice$ gdaladdo -r mode N084W180_x32_GLC.new.tif 2 4
0ERROR 1: N084W180_x32_GLC.new.tif:DumpModeDecode: Not enough data for scanline 0
ERROR 1: TIFFReadEncodedTile() failed.

ERROR 1: IReadBlock failed at X offset 0, Y offset 0
ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 0
...10...20...30...40...50...60...70...80...90...100 - done.
Overview building failed.
seth:gdal sprice$ gdaladdo -r near N084W180_x32_GLC.new.tif 2 4
0ERROR 1: N084W180_x32_GLC.new.tif:DumpModeDecode: Not enough data for scanline 0
ERROR 1: TIFFReadEncodedTile() failed.

ERROR 1: IReadBlock failed at X offset 0, Y offset 0
ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 0
...10...20...30...40...50...60...70...80...90...100 - done.
Overview building failed.
seth:gdal sprice$ gdal_translate N084W180_x32_GLC.tif N084W180_x32_GLC.new.tif
Input file size is 21600, 21600
0...10...20...30...40...50...60...70...80...90...100 - done.
seth:gdal sprice$ gdaladdo -r near N084W180_x32_GLC.new.tif 2 4
0...Warning 1: TIFFReadDirectory:N084W180_x32_GLC.new.tif: Wrong "StripByteCounts" field, ignoring and calculating from imagelength
Warning 1: TIFFReadDirectory:N084W180_x32_GLC.new.tif: Wrong "StripByteCounts" field, ignoring and calculating from imagelength
ERROR 1: N084W180_x32_GLC.new.tif:DumpModeDecode: Not enough data for scanline 0
ERROR 1: TIFFReadEncodedTile() failed.

ERROR 1: IReadBlock failed at X offset 0, Y offset 4
ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 4
10...20...30...40...50...60...70...80...90...100 - done.
Overview building failed.

10/26/08 12:00:00 changed by rouault

what do you call 'TIFF stable branch' ? 3.9 or 4.0 ? It is possible that some fixes have gone to libtiff 4.0 and not 3.9.

10/26/08 12:08:03 changed by sprice

The 3.9 branch. Is the 4.0 branch more stable now-a-days?

File too large for attachment, so I've uploaded it here: http://pricepages.org/temp/N084W180_x32_GLC.tif.gz

10/26/08 12:11:38 changed by rouault

The current internal libtiff version in GDAL is fetched from libtiff 4.0 CVS. It is fairly stable and nearly in a releasable state, though there are some remaining corner issues. I've used it in production for months.

Have you tried running your test with GDAL configured to use internal libtiff ?(./configure --with-libtiff=internal)

10/26/08 13:17:22 changed by sprice

Didn't work on a file that had perviously been corrupted, but otherwise works as expected with the internal tiff library.

11/22/08 11:13:07 changed by antonio

As a workaround for this issue you can use the 

--config USE_RRD YES

option. Unfortunately it doesn't seems to work for complex datasets: 

ERROR 6: Data type CInt16 not supported by Erdas Imagine (HFA) format.

01/25/09 13:37:40 changed by warmerdam

  • cc changed from mloskot to warmerdam.
  • owner changed from warmerdam to chaitanya.
  • milestone set to 1.6.1.

Chaitanya,

Could you try and reproduce this error compile GDAL trunk against the libtiff 3.9.0 beta release? If it does occur there, try again to see if you can reproduce it with libtiff 3.9.0 from the libtiff CVS repository (details on the libtiff site)? If the problem is still occuring with the libtiff 3.9 branch then we will need to dig into the details and fix it for libtiff 3.9 final release (coming soon).

01/26/09 22:31:33 changed by chaitanya

  • status changed from new to assigned.

02/01/09 09:59:31 changed by chaitanya

I compiled GDAL trunk against tiff-3.9.0beta.zip from ftp://ftp.remotesensing.org/pub/libtiff/ It worked. It also worked with the libtiff from the GDAL trunk. 

C:\warmerda\bld\bin>gdal_translate.exe N084W180_x32_GLC.tif N084W180_x32_GLC.new.tif
Input file size is 21600, 21600
0...10...20...30...40...50...60...70...80...90...100 - done.

C:\warmerda\bld\bin>gdaladdo.exe -r mode N084W180_x32_GLC.new.tif 2 4
0...10...20...30...40...50...60...70...80...90...100 - done.

C:\warmerda\bld\bin>gdaladdo.exe -r near N084W180_x32_GLC.new.tif 2 4
0...10...20...30...40...50...60...70...80...90...100 - done.

I couldn't reproduce the error either way. FYI, I tested this on Intel Pentium 4, 2.4GHz processor.

02/01/09 13:03:33 changed by warmerdam

  • status changed from assigned to closed.
  • resolution set to worksforme.

OK, things seem fixed up adequately.

Thanks