Ticket #2486 (closed defect: worksforme)

Opened 5 years ago

Last modified 4 years ago

IReadBlock failure in gdaladdo

Reported by: antonio Owned by: chaitanya
Priority: low Milestone: 1.6.1
Component: GDAL_Raster Version: 1.5.2
Severity: normal Keywords: GTiff
Cc: warmerdam

Description

I'm using GDAL 1.5.2 on debian sid. Trying the following: 

$ gdal_translate -of VRT HDF5:test9001.h5://S01/SBI test9001.h5.vrt
$ gdaladdo -r average test9001.h5.vrt 2 4

I get an IReadBlock failure: 

0...10...20...30...40...50...60...70...80...90..
Warning 1: TIFFReadDirectory:test9001.h5.vrt.ovr: Wrong "StripByteCounts" field, ignoring and calculating from imagelength
Warning 1: TIFFReadDirectory:test9001.h5.vrt.ovr: Wrong "StripByteCounts" field, ignoring and calculating from imagelength
ERROR 1: test9001.h5.vrt.ovr:DumpModeDecode: Not enough data for scanline 0
ERROR 1: TIFFReadEncodedTile() failed.

ERROR 1: IReadBlock failed at X offset 0, Y offset 15
ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 15
ERROR 1: test9001.h5.vrt.ovr:DumpModeDecode: Not enough data for scanline 0
ERROR 1: TIFFReadEncodedTile() failed.

[CUT]

ERROR 1: IReadBlock failed at X offset 1, Y offset 16
ERROR 1: GetBlockRef failed at X block offset 1, Y block offset 16
ERROR 1: test9001.h5.vrt.ovr:DumpModeDecode: Not enough data for scanline 128
ERROR 1: TIFFReadEncodedTile() failed.

More than 1000 errors or warnings have been reported. No more will be reported from now.
100 - done.

The problem seems do be related to the "-r average" option.

Attached the test files.

Attachments

test9001.h5.bz2 Download (279 bytes) - added by antonio 5 years ago.

Change History

Changed 5 years ago by antonio

follow-up: ↓ 2   Changed 5 years ago by warmerdam

  • cc warmerdam added
  • keywords GTiff added
  • component changed from default to GDAL_Raster
  • owner changed from warmerdam to mloskot
  • priority changed from normal to high

Mateusz,

Could you try and reproduce this?

Antonio,

Could you establish what version of libtiff is being used on debian sid? I suspect this is a problem already fixed in libtiff but that sid is using a somewhat older version of libtiff without the fix. Also, please confirm you are using the pre-packaged GDAL for Debian that uses the system libtiff, not one you built yourself that might be using an internal libtiff.

If the problem is just an old libtiff, I don't think there will be anything we can do.

in reply to: ↑ 1   Changed 5 years ago by antonio

Replying to warmerdam:

Mateusz, Could you try and reproduce this? Antonio, Could you establish what version of libtiff is being used on debian sid? I suspect this is a problem already fixed in libtiff but that sid is using a somewhat older version of libtiff without the fix. Also, please confirm you are using the pre-packaged GDAL for Debian that uses the system libtiff, not one you built yourself that might be using an internal libtiff.

Yes, I'm using standard Debian packages:

  • libgdal1-1.5.0 1.5.2-2 (unstable) --> depends on lintiff4
  • gdal-bin 1.5.2-2 (unstable)
  • libtiff4 3.8.2-10 (unstable)

  Changed 5 years ago by mloskot

  • status changed from new to assigned

Fellows,

Tonight, I've only access to Windows machine. I'm back to my regular desk tomorrow and will work on this issue.

  Changed 5 years ago by mloskot

Antonio, Frank,

I've reproduced this problem on Debian Sid (unstable) using GDAL and libtiff installed from packages.

I also made tests on Ubuntu 7.04 with current version of GDAL from SVN trunk configured with option --with-libtiff=internal and HDF5 support from Debian package libhdf5-serial-dev (1.6.5). Here are my results:

  • check HDF5 support 
    $ export CPL_DEBUG=ON

$ gdalinfo test9001.h5 
GDAL: GDALOpen(test9001.h5) succeeds as HDF5.
Driver: HDF5/Hierarchical Data Format Release 5
Files: test9001.h5
Size is 512, 512
Coordinate System is `'
Subdatasets:
  SUBDATASET_0_NAME=HDF5:"test9001.h5"://S01/SBI
  SUBDATASET_0_DESC=[9001x9001] //S01/SBI (8-bit unsigned integer)
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0,  512.0)
Upper Right (  512.0,    0.0)
Lower Right (  512.0,  512.0)
Center      (  256.0,  256.0)
GDAL: GDALClose(test9001.h5)
GDAL: GDALDeregister_GTiff() called.
  • translate to VRT 
    $ gdal_translate -of VRT HDF5:test9001.h5://S01/SBI test9001.h5.vrt
GDAL: GDALOpen(HDF5:test9001.h5://S01/SBI) succeeds as HDF5Image.
Input file size is 9001, 9001
GDAL: GDALClose(test9001.h5.vrt)
GDAL: GDALClose(HDF5:test9001.h5://S01/SBI)
GDAL: GDALDeregister_GTiff() called.
  • generate overviews with gdaladdo. No IReadBlock failures occurred, only warning about potential trashing. 
    $ gdaladdo -r average test9001.h5.vrt 2 4
GDAL: GDALOpen(test9001.h5.vrt) succeeds as VRT.
GTiff: Opened 2251x2251 overview.

GDAL: GDALOpen(test9001.h5.vrt.ovr) succeeds as GTiff.
0GDAL: GDALOpen(HDF5:test9001.h5://S01/SBI) succeeds as HDF5Image.
GTiff: SetDirectory(              8)
...10...20...30...40...50...60...70...GDAL: Potential thrashing on band 1 of test9001.h5.vrt.ovr.
GTiff: SetDirectory(          10538)
80GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
.GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
.GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
.GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
90GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
.GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
.GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
.GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
GTiff: SetDirectory(              8)
GTiff: SetDirectory(          10538)
100 - done.
GDAL: GDALClose()
GTiff: SetDirectory(              8)
GDAL: GDALClose(test9001.h5.vrt.ovr)
GDAL: 2592 block reads on 1296 block band 1 of test9001.h5.vrt.ovr.
GTiff: Opened 2251x2251 overview.

GDAL: GDALOpen(test9001.h5.vrt.ovr) succeeds as GTiff.
GDAL: GDALClose(test9001.h5.vrt)
GDAL: GDALClose(HDF5:test9001.h5://S01/SBI)
GDAL: GDALClose(HDF5:test9001.h5://S01/SBI)
GDAL: GDALClose()
GTiff: SetDirectory(              8)
GDAL: GDALClose(test9001.h5.vrt.ovr)
GDAL: GDALDeregister_GTiff() called.
  • inspecting generated overview 
    $ gdalinfo test9001.h5.vrt.ovr 
GTiff: Opened 2251x2251 overview.

GDAL: GDALOpen(test9001.h5.vrt.ovr) succeeds as GTiff.
Driver: GTiff/GeoTIFF
Files: test9001.h5.vrt.ovr
Size is 4501, 4501
Coordinate System is `'
Image Structure Metadata:
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0, 4501.0)
Upper Right ( 4501.0,    0.0)
Lower Right ( 4501.0, 4501.0)
Center      ( 2250.5, 2250.5)
Band 1 Block=128x128 Type=UInt16, ColorInterp=Gray
  Overviews: 2251x2251
GDAL: GDALClose()
GTiff: SetDirectory(              8)
GDAL: GDALClose(test9001.h5.vrt.ovr)
GDAL: GDALDeregister_GTiff() called.

Observing no crash of gdaladdo above, I'm inclined to support Frank's opinion the problem is in libtif version 3.8.2 used on Sid. Frank?

  Changed 5 years ago by warmerdam

  • cc mloskot added; warmerdam removed
  • owner changed from mloskot to warmerdam
  • status changed from assigned to new

I'm going to try this with 1.5 branch and libtiff internal vs. libtiff 3.8.2.

  Changed 5 years ago by warmerdam

  • priority changed from high to low
  • milestone 1.5.3 deleted

I was unable to briefly reproduce the problem but I didn't actually build libtiff 3.8.2 - I just used an old one that was installed on my system.

I'm dropping the milestone for this ticket since I believe it is a libtiff issue that has already been fixed.

  Changed 5 years ago by sprice

I almost opened a new ticket, but this looks like the same issue.

I'm running the latest code from GDAL trunk and TIFF stable branch, and I'm getting the same issue. In the code sample below, I first generate a fresh GeoTIFF. Then I try to add overviews in a few different manners. Note that the error is different between the first attempt and the second. The next attempts use nearest resampling, but it still fails each time.

I'll attach the file I'm using. 

seth:gdal sprice$ gdal_translate N084W180_x32_GLC.tif N084W180_x32_GLC.new.tif
Input file size is 21600, 21600
0...10...20...30...40...50...60...70...80...90...100 - done.
seth:gdal sprice$ gdaladdo -r mode N084W180_x32_GLC.new.tif 2 4
0...Warning 1: TIFFReadDirectory:N084W180_x32_GLC.new.tif: Wrong "StripByteCounts" field, ignoring and calculating from imagelength
Warning 1: TIFFReadDirectory:N084W180_x32_GLC.new.tif: Wrong "StripByteCounts" field, ignoring and calculating from imagelength
ERROR 1: N084W180_x32_GLC.new.tif:DumpModeDecode: Not enough data for scanline 0
ERROR 1: TIFFReadEncodedTile() failed.

ERROR 1: IReadBlock failed at X offset 0, Y offset 4
ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 4
10...20...30...40...50...60...70...80...90...100 - done.
Overview building failed.
seth:gdal sprice$ gdaladdo -r mode N084W180_x32_GLC.new.tif 2 4
0ERROR 1: N084W180_x32_GLC.new.tif:DumpModeDecode: Not enough data for scanline 0
ERROR 1: TIFFReadEncodedTile() failed.

ERROR 1: IReadBlock failed at X offset 0, Y offset 0
ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 0
...10...20...30...40...50...60...70...80...90...100 - done.
Overview building failed.
seth:gdal sprice$ gdaladdo -r near N084W180_x32_GLC.new.tif 2 4
0ERROR 1: N084W180_x32_GLC.new.tif:DumpModeDecode: Not enough data for scanline 0
ERROR 1: TIFFReadEncodedTile() failed.

ERROR 1: IReadBlock failed at X offset 0, Y offset 0
ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 0
...10...20...30...40...50...60...70...80...90...100 - done.
Overview building failed.
seth:gdal sprice$ gdal_translate N084W180_x32_GLC.tif N084W180_x32_GLC.new.tif
Input file size is 21600, 21600
0...10...20...30...40...50...60...70...80...90...100 - done.
seth:gdal sprice$ gdaladdo -r near N084W180_x32_GLC.new.tif 2 4
0...Warning 1: TIFFReadDirectory:N084W180_x32_GLC.new.tif: Wrong "StripByteCounts" field, ignoring and calculating from imagelength
Warning 1: TIFFReadDirectory:N084W180_x32_GLC.new.tif: Wrong "StripByteCounts" field, ignoring and calculating from imagelength
ERROR 1: N084W180_x32_GLC.new.tif:DumpModeDecode: Not enough data for scanline 0
ERROR 1: TIFFReadEncodedTile() failed.

ERROR 1: IReadBlock failed at X offset 0, Y offset 4
ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 4
10...20...30...40...50...60...70...80...90...100 - done.
Overview building failed.

  Changed 5 years ago by rouault

what do you call 'TIFF stable branch' ? 3.9 or 4.0 ? It is possible that some fixes have gone to libtiff 4.0 and not 3.9.

  Changed 5 years ago by sprice

The 3.9 branch. Is the 4.0 branch more stable now-a-days?

File too large for attachment, so I've uploaded it here:  http://pricepages.org/temp/N084W180_x32_GLC.tif.gz

  Changed 5 years ago by rouault

The current internal libtiff version in GDAL is fetched from libtiff 4.0 CVS. It is fairly stable and nearly in a releasable state, though there are some remaining corner issues. I've used it in production for months.

Have you tried running your test with GDAL configured to use internal libtiff ?(./configure --with-libtiff=internal)

  Changed 5 years ago by sprice

Didn't work on a file that had perviously been corrupted, but otherwise works as expected with the internal tiff library.

  Changed 5 years ago by antonio

As a workaround for this issue you can use the 

--config USE_RRD YES

option. Unfortunately it doesn't seems to work for complex datasets: 

ERROR 6: Data type CInt16 not supported by Erdas Imagine (HFA) format.

  Changed 4 years ago by warmerdam

  • cc warmerdam added; mloskot removed
  • owner changed from warmerdam to chaitanya
  • milestone set to 1.6.1

Chaitanya,

Could you try and reproduce this error compile GDAL trunk against the libtiff 3.9.0 beta release? If it does occur there, try again to see if you can reproduce it with libtiff 3.9.0 from the libtiff CVS repository (details on the libtiff site)? If the problem is still occuring with the libtiff 3.9 branch then we will need to dig into the details and fix it for libtiff 3.9 final release (coming soon).

  Changed 4 years ago by chaitanya

  • status changed from new to assigned

  Changed 4 years ago by chaitanya

I compiled GDAL trunk against tiff-3.9.0beta.zip from  ftp://ftp.remotesensing.org/pub/libtiff/ It worked. It also worked with the libtiff from the GDAL trunk. 

C:\warmerda\bld\bin>gdal_translate.exe N084W180_x32_GLC.tif N084W180_x32_GLC.new.tif
Input file size is 21600, 21600
0...10...20...30...40...50...60...70...80...90...100 - done.

C:\warmerda\bld\bin>gdaladdo.exe -r mode N084W180_x32_GLC.new.tif 2 4
0...10...20...30...40...50...60...70...80...90...100 - done.

C:\warmerda\bld\bin>gdaladdo.exe -r near N084W180_x32_GLC.new.tif 2 4
0...10...20...30...40...50...60...70...80...90...100 - done.

I couldn't reproduce the error either way. FYI, I tested this on Intel Pentium 4, 2.4GHz processor.

  Changed 4 years ago by warmerdam

  • status changed from assigned to closed
  • resolution set to worksforme

OK, things seem fixed up adequately.

Thanks

Note: See TracTickets for help on using tickets.