Opened 10 years ago

Closed 10 years ago

Last modified 9 years ago

#3539 closed defect (fixed)

Bad external JPEG compressed .ovr

Reported by: gaopeng Owned by: warmerdam
Priority: normal Milestone: 1.8.0
Component: GDAL_Raster Version: 1.6.1
Severity: normal Keywords: gdaladdo gtiff
Cc:

Description

Here is what I can see:

Building an external .ovr for a tiff image fails with the following errors. The failed attemp left a band ovr file. A new external .ovr is created in the proxy folder (it's fine). The left over .ovr is opened as the overviews, and it's blank.

Test data: ftp://GDAL@ftp.esri.com/090162.tif

ERROR 1: C:\Data\090162.tif.ovr:No space to read TIFF directory
ERROR 1: TIFFReadDirectory:Failed to read directory at offset 8846
ERROR 1: TIFFWriteEncodedStrip:Must set "ImageWidth" before writing data
ERROR 1: TIFFWriteEncodedStrip:Must set "ImageWidth" before writing data
ERROR 1: TIFFWriteEncodedStrip:Must set "ImageWidth" before writing data
ERROR 1: TIFFWriteEncodedStrip:Must set "ImageWidth" before writing data
ERROR 1: TIFFWriteEncodedStrip:Must set "ImageWidth" before writing data
ERROR 1: TIFFWriteEncodedStrip:Must set "ImageWidth" before writing data
ERROR 1: TIFFWriteEncodedStrip:Must set "ImageWidth" before writing data
ERROR 1: TIFFWriteEncodedStrip:Must set "ImageWidth" before writing data
ERROR 1: TIFFWriteEncodedStrip:Must set "ImageWidth" before writing data
ERROR 1: C:\Data\090162.tif.ovr:No space to read TIFF directory
ERROR 1: TIFFReadDirectory:Failed to read directory at offset 8846
ERROR 1: IReadBlock failed at X offset 0, Y offset 3
ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 3
ERROR 1: _TIFFVGetField:C:\Data\090162.tif.ovr: Invalid pseudo-tag "" (not supported by codec)
ERROR 1: C:\Data\090162.tif.ovr:No space to read TIFF directory
ERROR 1: TIFFReadDirectory:Failed to read directory at offset 8846
ERROR 1: TIFFWriteEncodedStrip:Must set "ImageWidth" before writing data
ERROR 6: WriteBlock() not supported for this dataset.
ERROR 1: C:\Data\090162.tif.ovr:No space to read TIFF directory
ERROR 1: TIFFReadDirectory:Failed to read directory at offset 12194
ERROR 1: TIFFWriteEncodedStrip:Must set "ImageWidth" before writing data
ERROR 6: WriteBlock() not supported for this dataset.

Change History (17)

comment:1 Changed 10 years ago by warmerdam

Keywords: gdaladdo gtiff added
Owner: changed from Warmerdam to warmerdam
Status: newassigned

I have downloaded the .tif file in question and run the command "gdaladdo 090162.tif 2 4 8" from the 1.6-esri branch with the base tiff file set to read-only and an .ovr was properly created with no apparent problems.

Can you be more specific about how I might reproduce this problem?

comment:2 Changed 10 years ago by gaopeng

The repro steps I have is for ArcGIS and is not useful for you. Two things ArcGIS does differently, it uses JPEG compression, and 4.

I will also see if we can repro this using gdaladdo.

comment:3 Changed 10 years ago by gaopeng

Here is a command line to repro:

C:\Data>gdaladdo --config COMPRESS_OVERVIEW JPEG --config COMPRESS_OVERVIEW JPEG
 --config JPEG_QUALITY_OVERVIEW 75 --config SIMPLE_JPEG_MAGIC YES --config BIGTI
FF_OVERVIEW IF_SAFER -ro 090162.tif 4 8 16 32 64
0..ERROR 1: 090162.tif.ovr:No space to read TIFF directory
ERROR 1: TIFFReadDirectory:Failed to read directory at offset 8846
ERROR 1: IReadBlock failed at X offset 0, Y offset 2
ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 2
ERROR 1: 090162.tif.ovr:No space to read TIFF directory
ERROR 1: TIFFReadDirectory:Failed to read directory at offset 8846
ERROR 1: TIFFWriteEncodedStrip:Must set "ImageWidth" before writing data
ERROR 6: WriteBlock() not supported for this dataset.
ERROR 1: 090162.tif.ovr:No space to read TIFF directory
ERROR 1: TIFFReadDirectory:Failed to read directory at offset 12194
ERROR 1: TIFFWriteEncodedStrip:Must set "ImageWidth" before writing data
ERROR 6: WriteBlock() not supported for this dataset.
.10...20...30...40...50...60...70...80...90...100 - done.
Overview building failed.

comment:4 Changed 10 years ago by gaopeng

The test is done using 1.6-esri build.

comment:5 Changed 10 years ago by warmerdam

Problem reproduced. I'm digging further into it now.

comment:6 Changed 10 years ago by warmerdam

I have backported some logic to ensure that directories are properly migrated when then need to be rewritten due to tag change. In this case the tag change is the deferred setting of the jpeg tables when the first image block is written.

I also modified the jpegtables logic so it is not redone each time setup is called as this triggered lots of unnecessary directory migrations wasting space and time. This aspect does not yet exist upstream and I'll need to apply it there (it is in tif_jpeg.c).

The changes are currently in 1.6-esri (r19483).

comment:7 Changed 10 years ago by warmerdam

Milestone: 1.8.0
Resolution: fixed
Status: assignedclosed

I have applied the prepare-jpeg tables change in libtiff CVS head, and I have applied the adapt-to-directory-move in FlushDirectory?() in GDAL trunk (r19484).

Arguably these fixes should be migrated into 1.7, but I'm somewhat hesitant.

comment:8 Changed 10 years ago by gaopeng

Resolution: fixed
Status: closedreopened

Creating JPEG compressed overviews for a 16bit tiff fails with the following error:

:\Data>gdaladdo --config COMPRESS_OVERVIEW JPEG --config JPEG_QUALITY_OVERVIEW
5 --config SIMPLE_JPEG_MAGIC YES --config BIGTIFF_OVERVIEW IF_SAFER -ro po_2849
3_grn_0000000.tif 4 8 16 32 64
...10...20...30...40...50.ERROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed
for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
.ERROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
0ERROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
.ERROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
70ERROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
.ERROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
ERROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
0.ERROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
.ERROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
0...ERROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
00 - done.
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 6: WriteBlock() not supported for this dataset.
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 6: WriteBlock() not supported for this dataset.
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
RROR 6: WriteBlock() not supported for this dataset.

comment:9 Changed 10 years ago by gaopeng

I made a fix to force 12 bit if compession is JPEG, and nBitsPerPixel is 16, in gt_overview.cpp. The test was run in 1.6-esri build.

    if( nCompression == COMPRESSION_JPEG && nBitsPerPixel == 16 )
        nBitsPerPixel = 12;

This fixed some cases, but uncoverd the following error. This looks similar to the problem fixed eailier for 8bit case.


C:\Data>gdaladdo --config COMPRESS_OVERVIEW JPEG --config JPEG_QUALITY_OVERVIEW
90 --config SIMPLE_JPEG_MAGIC YES --config BIGTIFF_OVERVIEW IF_SAFER -r average
-clean -ro 09MAY17191500-M2AS_R2C2-052188944090_01_P001.TIF 4 8 16
Warning 5: ***nBitsPerPixel=27055826.
0...10...20...30...40.ERROR 1: JPEGLib:Not a JPEG file: starts with 0x00 0x00
ERROR 1: JPEGSetupDecode:Bogus JPEGTables field
ERROR 1: JPEGLib:Not a JPEG file: starts with 0x00 0x00
ERROR 1: TIFFReadEncodedTile() failed.
ERROR 1: IReadBlock failed at X offset 0, Y offset 0
ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 0
..ERROR 1: JPEGLib:Not a JPEG file: starts with 0x00 0x00
ERROR 1: JPEGSetupDecode:Bogus JPEGTables field
ERROR 1: JPEGLib:Not a JPEG file: starts with 0x00 0x00
ERROR 1: TIFFReadEncodedTile() failed.
ERROR 1: IReadBlock failed at X offset 0, Y offset 0
ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 0
50...60...ERROR 1: JPEGLib:Not a JPEG file: starts with 0x00 0x00
ERROR 1: JPEGSetupDecode:Bogus JPEGTables field
ERROR 1: JPEGLib:Not a JPEG file: starts with 0x00 0x00
ERROR 1: TIFFReadEncodedTile() failed.
ERROR 1: IReadBlock failed at X offset 0, Y offset 0
ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 0
70.ERROR 1: JPEGLib:Not a JPEG file: starts with 0x00 0x00
ERROR 1: JPEGSetupDecode:Bogus JPEGTables field
ERROR 1: JPEGLib:Not a JPEG file: starts with 0x00 0x00
ERROR 1: TIFFReadEncodedTile() failed.
ERROR 1: IReadBlock failed at X offset 0, Y offset 0
ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 0
..80...90.ERROR 1: JPEGLib:Not a JPEG file: starts with 0x00 0x00
ERROR 1: JPEGSetupDecode:Bogus JPEGTables field
ERROR 1: JPEGLib:Not a JPEG file: starts with 0x00 0x00
ERROR 1: TIFFReadEncodedTile() failed.
ERROR 1: IReadBlock failed at X offset 0, Y offset 0
ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 0
..ERROR 1: JPEGLib:Not a JPEG file: starts with 0x00 0x00
ERROR 1: JPEGSetupDecode:Bogus JPEGTables field
ERROR 1: JPEGLib:Not a JPEG file: starts with 0x00 0x00
ERROR 1: TIFFReadEncodedTile() failed.
ERROR 1: IReadBlock failed at X offset 0, Y offset 0
ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 0

comment:10 Changed 10 years ago by warmerdam

Resolution: fixed
Status: reopenedclosed

I have incorporated the 16->12 bit change proposed above in trunk (r19621) and 1.6-esri (r19620).

I have isolated and corrected the problem with the file becoming corrupt to use of memcpy() instead of memcmp() in tif_jpeg.c which is fixed in 1.6-esri (r19619). This was also fixed upstream in libtiff, and the libtiff fix brought back into trunk (r19618).

I am now able to successfully build this 12bit overview.

Let me know if the problem persists at your end.

PS. Make sure you force a clean build - at least down in gdal/frmts/gtiff. The 12bit code includes tif_jpeg.c but this dependency is not recognised by the makefiles.

comment:11 Changed 10 years ago by gaopeng

Resolution: fixed
Status: closedreopened

I no long get any errors. However, the generated overviews have a bad (redish) block at top-left corner. The overviews, 09MAY17191500-M2AS_R2C2-052188944090_01_P001.TIF.ovr.tif, is at ftp://GDAL@ftp.esri.com

comment:12 Changed 10 years ago by warmerdam

I am also seeing serious problems with the overviews being generated in 1.6-esri and trunk. I am digging.

comment:13 Changed 10 years ago by warmerdam

I have found that the problem does not occur with nearest neighbour resampling. The difference seems to be that averaging uses GDALRegenerateCascadingOverviews() while nearest just computes all levels directly from the full resolution imagery. The cascading overviews involves re-reading from the recently written new overviews and the values seem to come back altered for reasons that are not clear in some situations (bands 2, 3, 4 and only after the first strip).

A work around that could be applied immediately is to #ifdef out the call to GDALRegenerateCascadingOverviews() in gdal/gcore/overview.cpp. However, I am continuing to dig to try and come up with a proper fix.

comment:14 Changed 10 years ago by warmerdam

I have established that the overview is being written in some fashion that the order of reading blocks matters. With the overview file http://home.gdal.org/tmp/bug3539.tif I find that the following program produces wrong results (pixel around 1300) reading the second block in the second band unless the first block is read first in which case the correct value (around 300) is returned.

#include "gdal_priv.h"

int main( int argc, char ** argv )

{
    GInt16 anBlock[128*128];

    GDALAllRegister();

    GDALDataset *poDS = (GDALDataset *) GDALOpen( "bug3539.tif", GA_ReadOnly );
    GDALRasterBand *poBand = poDS->GetRasterBand(2);

    if( argc > 1 && strcmp(argv[1],"-pre") == 0 )
    {
        printf( "Preread first block.\n" );
        poBand->RasterIO( GF_Read, 0, 0, 128, 128, 
                          anBlock, 128, 128, GDT_Int16, 
                          0, 0 );
    }

    poBand->RasterIO( GF_Read, 128, 0, 128, 128, 
                      anBlock, 128, 128, GDT_Int16, 
                      0, 0 );

    printf( "band2[0,128] = %d\n", anBlock[0] );

    delete poDS;
}

I am guessing there is some sort of jpeg table written out in some blocks and not others and a block with it must be read before other blocks in order for decoding to work properly. Note that this pattern occurs on bands 2-4 but not on band 1.

I continue to dig.

It seems likely that this problem will affect other non-linear access to 12bit jpeg compressed tiff files though I have not confirmed that. The issue might also be specific to how the overviews are written (with jumping around from overview level to overview level resetting the jpeg encoder machinery).

comment:15 Changed 10 years ago by warmerdam

Resolution: fixed
Status: reopenedclosed

I believe I have traced this down to problems propagating non-default jpeg quality values down through libtiff when the current directory changes in the midst of encoding of tiles. The fix is upstream in libtiff (tif_jpeg.c), and I have incorporated it in 1.6-esri (r19632) and also trunk (r19633).

Please confirm that overview generation is working now. Improperly created overviews will need to be deleted/recreated.

While the problem was encountered with 12bit overviews it seems to me that it could also occur with the more common 8bit overviews.

comment:16 Changed 10 years ago by gaopeng

Tested, and it works well. Thank you very much.

comment:17 Changed 9 years ago by Even Rouault

r20145 /trunk/gdal/frmts/gtiff/geotiff.cpp: GTiff: only apply changes done in FlushDirectory?() in r19484 for #3539 in the libtiff 4.X case only, as they cause regressions in the libtiff 3.X case

Note: See TracTickets for help on using tickets.