#3539 closed defect (fixed)
Bad external JPEG compressed .ovr
| Reported by: | gaopeng | Owned by: | warmerdam |
|---|---|---|---|
| Priority: | normal | Milestone: | 1.8.0 |
| Component: | GDAL_Raster | Version: | 1.6.1 |
| Severity: | normal | Keywords: | gdaladdo gtiff |
| Cc: |
Description
Here is what I can see:
Building an external .ovr for a tiff image fails with the following errors. The failed attemp left a band ovr file. A new external .ovr is created in the proxy folder (it's fine). The left over .ovr is opened as the overviews, and it's blank.
Test data: ftp://GDAL@ftp.esri.com/090162.tif
ERROR 1: C:\Data\090162.tif.ovr:No space to read TIFF directory ERROR 1: TIFFReadDirectory:Failed to read directory at offset 8846 ERROR 1: TIFFWriteEncodedStrip:Must set "ImageWidth" before writing data ERROR 1: TIFFWriteEncodedStrip:Must set "ImageWidth" before writing data ERROR 1: TIFFWriteEncodedStrip:Must set "ImageWidth" before writing data ERROR 1: TIFFWriteEncodedStrip:Must set "ImageWidth" before writing data ERROR 1: TIFFWriteEncodedStrip:Must set "ImageWidth" before writing data ERROR 1: TIFFWriteEncodedStrip:Must set "ImageWidth" before writing data ERROR 1: TIFFWriteEncodedStrip:Must set "ImageWidth" before writing data ERROR 1: TIFFWriteEncodedStrip:Must set "ImageWidth" before writing data ERROR 1: TIFFWriteEncodedStrip:Must set "ImageWidth" before writing data ERROR 1: C:\Data\090162.tif.ovr:No space to read TIFF directory ERROR 1: TIFFReadDirectory:Failed to read directory at offset 8846 ERROR 1: IReadBlock failed at X offset 0, Y offset 3 ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 3 ERROR 1: _TIFFVGetField:C:\Data\090162.tif.ovr: Invalid pseudo-tag "" (not supported by codec) ERROR 1: C:\Data\090162.tif.ovr:No space to read TIFF directory ERROR 1: TIFFReadDirectory:Failed to read directory at offset 8846 ERROR 1: TIFFWriteEncodedStrip:Must set "ImageWidth" before writing data ERROR 6: WriteBlock() not supported for this dataset. ERROR 1: C:\Data\090162.tif.ovr:No space to read TIFF directory ERROR 1: TIFFReadDirectory:Failed to read directory at offset 12194 ERROR 1: TIFFWriteEncodedStrip:Must set "ImageWidth" before writing data ERROR 6: WriteBlock() not supported for this dataset.
Change History (17)
comment:1 by , 13 years ago
| Keywords: | gdaladdo gtiff added |
|---|---|
| Owner: | changed from to |
| Status: | new → assigned |
comment:2 by , 13 years ago
The repro steps I have is for ArcGIS and is not useful for you. Two things ArcGIS does differently, it uses JPEG compression, and 4.
I will also see if we can repro this using gdaladdo.
comment:3 by , 13 years ago
Here is a command line to repro:
C:\Data>gdaladdo --config COMPRESS_OVERVIEW JPEG --config COMPRESS_OVERVIEW JPEG --config JPEG_QUALITY_OVERVIEW 75 --config SIMPLE_JPEG_MAGIC YES --config BIGTI FF_OVERVIEW IF_SAFER -ro 090162.tif 4 8 16 32 64 0..ERROR 1: 090162.tif.ovr:No space to read TIFF directory ERROR 1: TIFFReadDirectory:Failed to read directory at offset 8846 ERROR 1: IReadBlock failed at X offset 0, Y offset 2 ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 2 ERROR 1: 090162.tif.ovr:No space to read TIFF directory ERROR 1: TIFFReadDirectory:Failed to read directory at offset 8846 ERROR 1: TIFFWriteEncodedStrip:Must set "ImageWidth" before writing data ERROR 6: WriteBlock() not supported for this dataset. ERROR 1: 090162.tif.ovr:No space to read TIFF directory ERROR 1: TIFFReadDirectory:Failed to read directory at offset 12194 ERROR 1: TIFFWriteEncodedStrip:Must set "ImageWidth" before writing data ERROR 6: WriteBlock() not supported for this dataset. .10...20...30...40...50...60...70...80...90...100 - done. Overview building failed.
comment:6 by , 13 years ago
I have backported some logic to ensure that directories are properly migrated when then need to be rewritten due to tag change. In this case the tag change is the deferred setting of the jpeg tables when the first image block is written.
I also modified the jpegtables logic so it is not redone each time setup is called as this triggered lots of unnecessary directory migrations wasting space and time. This aspect does not yet exist upstream and I'll need to apply it there (it is in tif_jpeg.c).
The changes are currently in 1.6-esri (r19483).
comment:7 by , 13 years ago
| Milestone: | → 1.8.0 |
|---|---|
| Resolution: | → fixed |
| Status: | assigned → closed |
I have applied the prepare-jpeg tables change in libtiff CVS head, and I have applied the adapt-to-directory-move in FlushDirectory() in GDAL trunk (r19484).
Arguably these fixes should be migrated into 1.7, but I'm somewhat hesitant.
comment:8 by , 13 years ago
| Resolution: | fixed |
|---|---|
| Status: | closed → reopened |
Creating JPEG compressed overviews for a 16bit tiff fails with the following error:
:\Data>gdaladdo --config COMPRESS_OVERVIEW JPEG --config JPEG_QUALITY_OVERVIEW 5 --config SIMPLE_JPEG_MAGIC YES --config BIGTIFF_OVERVIEW IF_SAFER -ro po_2849 3_grn_0000000.tif 4 8 16 32 64 ...10...20...30...40...50.ERROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG .ERROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG 0ERROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG .ERROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG 70ERROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG .ERROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG ERROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG 0.ERROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG .ERROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG 0...ERROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG 00 - done. RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 6: WriteBlock() not supported for this dataset. RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 6: WriteBlock() not supported for this dataset. RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG RROR 6: WriteBlock() not supported for this dataset.
comment:9 by , 13 years ago
I made a fix to force 12 bit if compession is JPEG, and nBitsPerPixel is 16, in gt_overview.cpp. The test was run in 1.6-esri build.
if( nCompression == COMPRESSION_JPEG && nBitsPerPixel == 16 )
nBitsPerPixel = 12;
This fixed some cases, but uncoverd the following error. This looks similar to the problem fixed eailier for 8bit case.
C:\Data>gdaladdo --config COMPRESS_OVERVIEW JPEG --config JPEG_QUALITY_OVERVIEW 90 --config SIMPLE_JPEG_MAGIC YES --config BIGTIFF_OVERVIEW IF_SAFER -r average -clean -ro 09MAY17191500-M2AS_R2C2-052188944090_01_P001.TIF 4 8 16 Warning 5: ***nBitsPerPixel=27055826. 0...10...20...30...40.ERROR 1: JPEGLib:Not a JPEG file: starts with 0x00 0x00 ERROR 1: JPEGSetupDecode:Bogus JPEGTables field ERROR 1: JPEGLib:Not a JPEG file: starts with 0x00 0x00 ERROR 1: TIFFReadEncodedTile() failed. ERROR 1: IReadBlock failed at X offset 0, Y offset 0 ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 0 ..ERROR 1: JPEGLib:Not a JPEG file: starts with 0x00 0x00 ERROR 1: JPEGSetupDecode:Bogus JPEGTables field ERROR 1: JPEGLib:Not a JPEG file: starts with 0x00 0x00 ERROR 1: TIFFReadEncodedTile() failed. ERROR 1: IReadBlock failed at X offset 0, Y offset 0 ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 0 50...60...ERROR 1: JPEGLib:Not a JPEG file: starts with 0x00 0x00 ERROR 1: JPEGSetupDecode:Bogus JPEGTables field ERROR 1: JPEGLib:Not a JPEG file: starts with 0x00 0x00 ERROR 1: TIFFReadEncodedTile() failed. ERROR 1: IReadBlock failed at X offset 0, Y offset 0 ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 0 70.ERROR 1: JPEGLib:Not a JPEG file: starts with 0x00 0x00 ERROR 1: JPEGSetupDecode:Bogus JPEGTables field ERROR 1: JPEGLib:Not a JPEG file: starts with 0x00 0x00 ERROR 1: TIFFReadEncodedTile() failed. ERROR 1: IReadBlock failed at X offset 0, Y offset 0 ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 0 ..80...90.ERROR 1: JPEGLib:Not a JPEG file: starts with 0x00 0x00 ERROR 1: JPEGSetupDecode:Bogus JPEGTables field ERROR 1: JPEGLib:Not a JPEG file: starts with 0x00 0x00 ERROR 1: TIFFReadEncodedTile() failed. ERROR 1: IReadBlock failed at X offset 0, Y offset 0 ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 0 ..ERROR 1: JPEGLib:Not a JPEG file: starts with 0x00 0x00 ERROR 1: JPEGSetupDecode:Bogus JPEGTables field ERROR 1: JPEGLib:Not a JPEG file: starts with 0x00 0x00 ERROR 1: TIFFReadEncodedTile() failed. ERROR 1: IReadBlock failed at X offset 0, Y offset 0 ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 0
comment:10 by , 13 years ago
| Resolution: | → fixed |
|---|---|
| Status: | reopened → closed |
I have incorporated the 16->12 bit change proposed above in trunk (r19621) and 1.6-esri (r19620).
I have isolated and corrected the problem with the file becoming corrupt to use of memcpy() instead of memcmp() in tif_jpeg.c which is fixed in 1.6-esri (r19619). This was also fixed upstream in libtiff, and the libtiff fix brought back into trunk (r19618).
I am now able to successfully build this 12bit overview.
Let me know if the problem persists at your end.
PS. Make sure you force a clean build - at least down in gdal/frmts/gtiff. The 12bit code includes tif_jpeg.c but this dependency is not recognised by the makefiles.
comment:11 by , 13 years ago
| Resolution: | fixed |
|---|---|
| Status: | closed → reopened |
I no long get any errors. However, the generated overviews have a bad (redish) block at top-left corner. The overviews, 09MAY17191500-M2AS_R2C2-052188944090_01_P001.TIF.ovr.tif, is at ftp://GDAL@ftp.esri.com
comment:12 by , 13 years ago
I am also seeing serious problems with the overviews being generated in 1.6-esri and trunk. I am digging.
comment:13 by , 13 years ago
I have found that the problem does not occur with nearest neighbour resampling. The difference seems to be that averaging uses GDALRegenerateCascadingOverviews() while nearest just computes all levels directly from the full resolution imagery. The cascading overviews involves re-reading from the recently written new overviews and the values seem to come back altered for reasons that are not clear in some situations (bands 2, 3, 4 and only after the first strip).
A work around that could be applied immediately is to #ifdef out the call to GDALRegenerateCascadingOverviews() in gdal/gcore/overview.cpp. However, I am continuing to dig to try and come up with a proper fix.
comment:14 by , 13 years ago
I have established that the overview is being written in some fashion that the order of reading blocks matters. With the overview file http://home.gdal.org/tmp/bug3539.tif I find that the following program produces wrong results (pixel around 1300) reading the second block in the second band unless the first block is read first in which case the correct value (around 300) is returned.
#include "gdal_priv.h"
int main( int argc, char ** argv )
{
GInt16 anBlock[128*128];
GDALAllRegister();
GDALDataset *poDS = (GDALDataset *) GDALOpen( "bug3539.tif", GA_ReadOnly );
GDALRasterBand *poBand = poDS->GetRasterBand(2);
if( argc > 1 && strcmp(argv[1],"-pre") == 0 )
{
printf( "Preread first block.\n" );
poBand->RasterIO( GF_Read, 0, 0, 128, 128,
anBlock, 128, 128, GDT_Int16,
0, 0 );
}
poBand->RasterIO( GF_Read, 128, 0, 128, 128,
anBlock, 128, 128, GDT_Int16,
0, 0 );
printf( "band2[0,128] = %d\n", anBlock[0] );
delete poDS;
}
I am guessing there is some sort of jpeg table written out in some blocks and not others and a block with it must be read before other blocks in order for decoding to work properly. Note that this pattern occurs on bands 2-4 but not on band 1.
I continue to dig.
It seems likely that this problem will affect other non-linear access to 12bit jpeg compressed tiff files though I have not confirmed that. The issue might also be specific to how the overviews are written (with jumping around from overview level to overview level resetting the jpeg encoder machinery).
comment:15 by , 13 years ago
| Resolution: | → fixed |
|---|---|
| Status: | reopened → closed |
I believe I have traced this down to problems propagating non-default jpeg quality values down through libtiff when the current directory changes in the midst of encoding of tiles. The fix is upstream in libtiff (tif_jpeg.c), and I have incorporated it in 1.6-esri (r19632) and also trunk (r19633).
Please confirm that overview generation is working now. Improperly created overviews will need to be deleted/recreated.
While the problem was encountered with 12bit overviews it seems to me that it could also occur with the more common 8bit overviews.
comment:17 by , 13 years ago
r20145 /trunk/gdal/frmts/gtiff/geotiff.cpp: GTiff: only apply changes done in FlushDirectory() in r19484 for #3539 in the libtiff 4.X case only, as they cause regressions in the libtiff 3.X case
I have downloaded the .tif file in question and run the command "gdaladdo 090162.tif 2 4 8" from the 1.6-esri branch with the base tiff file set to read-only and an .ovr was properly created with no apparent problems.
Can you be more specific about how I might reproduce this problem?