Opened 16 years ago
Closed 15 years ago
#2486 closed defect (worksforme)
IReadBlock failure in gdaladdo
| Reported by: | antonio | Owned by: | chaitanya |
|---|---|---|---|
| Priority: | low | Milestone: | 1.6.1 |
| Component: | GDAL_Raster | Version: | 1.5.2 |
| Severity: | normal | Keywords: | GTiff |
| Cc: | warmerdam |
Description
I'm using GDAL 1.5.2 on debian sid. Trying the following:
$ gdal_translate -of VRT HDF5:test9001.h5://S01/SBI test9001.h5.vrt $ gdaladdo -r average test9001.h5.vrt 2 4
I get an IReadBlock failure:
0...10...20...30...40...50...60...70...80...90.. Warning 1: TIFFReadDirectory:test9001.h5.vrt.ovr: Wrong "StripByteCounts" field, ignoring and calculating from imagelength Warning 1: TIFFReadDirectory:test9001.h5.vrt.ovr: Wrong "StripByteCounts" field, ignoring and calculating from imagelength ERROR 1: test9001.h5.vrt.ovr:DumpModeDecode: Not enough data for scanline 0 ERROR 1: TIFFReadEncodedTile() failed. ERROR 1: IReadBlock failed at X offset 0, Y offset 15 ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 15 ERROR 1: test9001.h5.vrt.ovr:DumpModeDecode: Not enough data for scanline 0 ERROR 1: TIFFReadEncodedTile() failed. [CUT] ERROR 1: IReadBlock failed at X offset 1, Y offset 16 ERROR 1: GetBlockRef failed at X block offset 1, Y block offset 16 ERROR 1: test9001.h5.vrt.ovr:DumpModeDecode: Not enough data for scanline 128 ERROR 1: TIFFReadEncodedTile() failed. More than 1000 errors or warnings have been reported. No more will be reported from now. 100 - done.
The problem seems do be related to the "-r average" option.
Attached the test files.
Attachments (1)
Change History (17)
by , 16 years ago
| Attachment: | test9001.h5.bz2 added |
|---|
follow-up: 2 comment:1 by , 16 years ago
| Cc: | added |
|---|---|
| Component: | default → GDAL_Raster |
| Keywords: | GTiff added |
| Owner: | changed from to |
| Priority: | normal → high |
comment:2 by , 16 years ago
Replying to warmerdam:
Mateusz,
Could you try and reproduce this?
Antonio,
Could you establish what version of libtiff is being used on debian sid? I suspect this is a problem already fixed in libtiff but that sid is using a somewhat older version of libtiff without the fix. Also, please confirm you are using the pre-packaged GDAL for Debian that uses the system libtiff, not one you built yourself that might be using an internal libtiff.
Yes, I'm using standard Debian packages:
- libgdal1-1.5.0 1.5.2-2 (unstable) --> depends on lintiff4
- gdal-bin 1.5.2-2 (unstable)
- libtiff4 3.8.2-10 (unstable)
comment:3 by , 16 years ago
| Status: | new → assigned |
|---|
Fellows,
Tonight, I've only access to Windows machine. I'm back to my regular desk tomorrow and will work on this issue.
comment:4 by , 16 years ago
Antonio, Frank,
I've reproduced this problem on Debian Sid (unstable) using GDAL and libtiff installed from packages.
I also made tests on Ubuntu 7.04 with current version of GDAL from SVN trunk configured with option --with-libtiff=internal and HDF5 support from Debian package libhdf5-serial-dev (1.6.5). Here are my results:
- check HDF5 support
$ export CPL_DEBUG=ON $ gdalinfo test9001.h5 GDAL: GDALOpen(test9001.h5) succeeds as HDF5. Driver: HDF5/Hierarchical Data Format Release 5 Files: test9001.h5 Size is 512, 512 Coordinate System is `' Subdatasets: SUBDATASET_0_NAME=HDF5:"test9001.h5"://S01/SBI SUBDATASET_0_DESC=[9001x9001] //S01/SBI (8-bit unsigned integer) Corner Coordinates: Upper Left ( 0.0, 0.0) Lower Left ( 0.0, 512.0) Upper Right ( 512.0, 0.0) Lower Right ( 512.0, 512.0) Center ( 256.0, 256.0) GDAL: GDALClose(test9001.h5) GDAL: GDALDeregister_GTiff() called.
- translate to VRT
$ gdal_translate -of VRT HDF5:test9001.h5://S01/SBI test9001.h5.vrt GDAL: GDALOpen(HDF5:test9001.h5://S01/SBI) succeeds as HDF5Image. Input file size is 9001, 9001 GDAL: GDALClose(test9001.h5.vrt) GDAL: GDALClose(HDF5:test9001.h5://S01/SBI) GDAL: GDALDeregister_GTiff() called.
- generate overviews with gdaladdo. No IReadBlock failures occurred, only warning about potential trashing.
$ gdaladdo -r average test9001.h5.vrt 2 4 GDAL: GDALOpen(test9001.h5.vrt) succeeds as VRT. GTiff: Opened 2251x2251 overview. GDAL: GDALOpen(test9001.h5.vrt.ovr) succeeds as GTiff. 0GDAL: GDALOpen(HDF5:test9001.h5://S01/SBI) succeeds as HDF5Image. GTiff: SetDirectory( 8) ...10...20...30...40...50...60...70...GDAL: Potential thrashing on band 1 of test9001.h5.vrt.ovr. GTiff: SetDirectory( 10538) 80GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) .GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) .GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) .GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) 90GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) .GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) .GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) .GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) GTiff: SetDirectory( 8) GTiff: SetDirectory( 10538) 100 - done. GDAL: GDALClose() GTiff: SetDirectory( 8) GDAL: GDALClose(test9001.h5.vrt.ovr) GDAL: 2592 block reads on 1296 block band 1 of test9001.h5.vrt.ovr. GTiff: Opened 2251x2251 overview. GDAL: GDALOpen(test9001.h5.vrt.ovr) succeeds as GTiff. GDAL: GDALClose(test9001.h5.vrt) GDAL: GDALClose(HDF5:test9001.h5://S01/SBI) GDAL: GDALClose(HDF5:test9001.h5://S01/SBI) GDAL: GDALClose() GTiff: SetDirectory( 8) GDAL: GDALClose(test9001.h5.vrt.ovr) GDAL: GDALDeregister_GTiff() called.
- inspecting generated overview
$ gdalinfo test9001.h5.vrt.ovr GTiff: Opened 2251x2251 overview. GDAL: GDALOpen(test9001.h5.vrt.ovr) succeeds as GTiff. Driver: GTiff/GeoTIFF Files: test9001.h5.vrt.ovr Size is 4501, 4501 Coordinate System is `' Image Structure Metadata: INTERLEAVE=BAND Corner Coordinates: Upper Left ( 0.0, 0.0) Lower Left ( 0.0, 4501.0) Upper Right ( 4501.0, 0.0) Lower Right ( 4501.0, 4501.0) Center ( 2250.5, 2250.5) Band 1 Block=128x128 Type=UInt16, ColorInterp=Gray Overviews: 2251x2251 GDAL: GDALClose() GTiff: SetDirectory( 8) GDAL: GDALClose(test9001.h5.vrt.ovr) GDAL: GDALDeregister_GTiff() called.
Observing no crash of gdaladdo above, I'm inclined to support Frank's opinion the problem is in libtif version 3.8.2 used on Sid. Frank?
comment:5 by , 16 years ago
| Cc: | added; removed |
|---|---|
| Owner: | changed from to |
| Status: | assigned → new |
I'm going to try this with 1.5 branch and libtiff internal vs. libtiff 3.8.2.
comment:6 by , 16 years ago
| Milestone: | 1.5.3 |
|---|---|
| Priority: | high → low |
I was unable to briefly reproduce the problem but I didn't actually build libtiff 3.8.2 - I just used an old one that was installed on my system.
I'm dropping the milestone for this ticket since I believe it is a libtiff issue that has already been fixed.
comment:7 by , 16 years ago
I almost opened a new ticket, but this looks like the same issue.
I'm running the latest code from GDAL trunk and TIFF stable branch, and I'm getting the same issue. In the code sample below, I first generate a fresh GeoTIFF. Then I try to add overviews in a few different manners. Note that the error is different between the first attempt and the second. The next attempts use nearest resampling, but it still fails each time.
I'll attach the file I'm using.
seth:gdal sprice$ gdal_translate N084W180_x32_GLC.tif N084W180_x32_GLC.new.tif Input file size is 21600, 21600 0...10...20...30...40...50...60...70...80...90...100 - done. seth:gdal sprice$ gdaladdo -r mode N084W180_x32_GLC.new.tif 2 4 0...Warning 1: TIFFReadDirectory:N084W180_x32_GLC.new.tif: Wrong "StripByteCounts" field, ignoring and calculating from imagelength Warning 1: TIFFReadDirectory:N084W180_x32_GLC.new.tif: Wrong "StripByteCounts" field, ignoring and calculating from imagelength ERROR 1: N084W180_x32_GLC.new.tif:DumpModeDecode: Not enough data for scanline 0 ERROR 1: TIFFReadEncodedTile() failed. ERROR 1: IReadBlock failed at X offset 0, Y offset 4 ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 4 10...20...30...40...50...60...70...80...90...100 - done. Overview building failed. seth:gdal sprice$ gdaladdo -r mode N084W180_x32_GLC.new.tif 2 4 0ERROR 1: N084W180_x32_GLC.new.tif:DumpModeDecode: Not enough data for scanline 0 ERROR 1: TIFFReadEncodedTile() failed. ERROR 1: IReadBlock failed at X offset 0, Y offset 0 ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 0 ...10...20...30...40...50...60...70...80...90...100 - done. Overview building failed. seth:gdal sprice$ gdaladdo -r near N084W180_x32_GLC.new.tif 2 4 0ERROR 1: N084W180_x32_GLC.new.tif:DumpModeDecode: Not enough data for scanline 0 ERROR 1: TIFFReadEncodedTile() failed. ERROR 1: IReadBlock failed at X offset 0, Y offset 0 ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 0 ...10...20...30...40...50...60...70...80...90...100 - done. Overview building failed. seth:gdal sprice$ gdal_translate N084W180_x32_GLC.tif N084W180_x32_GLC.new.tif Input file size is 21600, 21600 0...10...20...30...40...50...60...70...80...90...100 - done. seth:gdal sprice$ gdaladdo -r near N084W180_x32_GLC.new.tif 2 4 0...Warning 1: TIFFReadDirectory:N084W180_x32_GLC.new.tif: Wrong "StripByteCounts" field, ignoring and calculating from imagelength Warning 1: TIFFReadDirectory:N084W180_x32_GLC.new.tif: Wrong "StripByteCounts" field, ignoring and calculating from imagelength ERROR 1: N084W180_x32_GLC.new.tif:DumpModeDecode: Not enough data for scanline 0 ERROR 1: TIFFReadEncodedTile() failed. ERROR 1: IReadBlock failed at X offset 0, Y offset 4 ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 4 10...20...30...40...50...60...70...80...90...100 - done. Overview building failed.
comment:8 by , 16 years ago
what do you call 'TIFF stable branch' ? 3.9 or 4.0 ? It is possible that some fixes have gone to libtiff 4.0 and not 3.9.
comment:9 by , 16 years ago
The 3.9 branch. Is the 4.0 branch more stable now-a-days?
File too large for attachment, so I've uploaded it here: http://pricepages.org/temp/N084W180_x32_GLC.tif.gz
comment:10 by , 16 years ago
The current internal libtiff version in GDAL is fetched from libtiff 4.0 CVS. It is fairly stable and nearly in a releasable state, though there are some remaining corner issues. I've used it in production for months.
Have you tried running your test with GDAL configured to use internal libtiff ?(./configure --with-libtiff=internal)
comment:11 by , 16 years ago
Didn't work on a file that had perviously been corrupted, but otherwise works as expected with the internal tiff library.
comment:12 by , 15 years ago
As a workaround for this issue you can use the
--config USE_RRD YES
option. Unfortunately it doesn't seems to work for complex datasets:
ERROR 6: Data type CInt16 not supported by Erdas Imagine (HFA) format.
comment:13 by , 15 years ago
| Cc: | added; removed |
|---|---|
| Milestone: | → 1.6.1 |
| Owner: | changed from to |
Chaitanya,
Could you try and reproduce this error compile GDAL trunk against the libtiff 3.9.0 beta release? If it does occur there, try again to see if you can reproduce it with libtiff 3.9.0 from the libtiff CVS repository (details on the libtiff site)? If the problem is still occuring with the libtiff 3.9 branch then we will need to dig into the details and fix it for libtiff 3.9 final release (coming soon).
comment:14 by , 15 years ago
| Status: | new → assigned |
|---|
comment:15 by , 15 years ago
I compiled GDAL trunk against tiff-3.9.0beta.zip from ftp://ftp.remotesensing.org/pub/libtiff/ It worked. It also worked with the libtiff from the GDAL trunk.
C:\warmerda\bld\bin>gdal_translate.exe N084W180_x32_GLC.tif N084W180_x32_GLC.new.tif Input file size is 21600, 21600 0...10...20...30...40...50...60...70...80...90...100 - done. C:\warmerda\bld\bin>gdaladdo.exe -r mode N084W180_x32_GLC.new.tif 2 4 0...10...20...30...40...50...60...70...80...90...100 - done. C:\warmerda\bld\bin>gdaladdo.exe -r near N084W180_x32_GLC.new.tif 2 4 0...10...20...30...40...50...60...70...80...90...100 - done.
I couldn't reproduce the error either way. FYI, I tested this on Intel Pentium 4, 2.4GHz processor.
comment:16 by , 15 years ago
| Resolution: | → worksforme |
|---|---|
| Status: | assigned → closed |
OK, things seem fixed up adequately.
Thanks
Mateusz,
Could you try and reproduce this?
Antonio,
Could you establish what version of libtiff is being used on debian sid? I suspect this is a problem already fixed in libtiff but that sid is using a somewhat older version of libtiff without the fix. Also, please confirm you are using the pre-packaged GDAL for Debian that uses the system libtiff, not one you built yourself that might be using an internal libtiff.
If the problem is just an old libtiff, I don't think there will be anything we can do.