libtiff refresh in GDAL 1.6.0dev trunk broke the ability to BuildOverviews() on a newly created GTiff

[Even Rouault]

I've tried the following python script :

import gdal
ds=gdal.GetDriverByName('GTiff').Create('tiled.tif',10000,10000,4)
ds.GetRasterBand(1).Fill(0)
ds.BuildOverviews('NEAR', overviewlist = [2])

(I think it is important that the size of the image must be big enough in comparison to the block cache so that actual data goes to the disk before the overview building. The fact that it is tiled and compressed doesn't change the behaviour)

On trunk, the scripts fails with the error messages reported in the previous emails. I also confirms that closing and reopening in update mode works around the error. On branches/1.5 it succeeds.

Then I imported frmt/gtiff/libtiff/* from a trunk tree into an 1.5 tree, compiled and the test fails And the opposite, I imported frmt/gtiff/libtiff/* from a 1.5 tree into a trunk tree, compiled and the test succeeds

So it is clearly a refresh from upstream libtiff in trunk since 1.5 that broke that test.

[Even Rouault]

By mixing versions of the libtiff files, I come to the conclusion that the regression comes fron an update in one or several of the following files : frmts/gtiff/libtiff/tif_dirread.c frmts/gtiff/libtiff/tif_dirwrite.c frmts/gtiff/libtiff/tif_flush.c

That doesn't help a lot since all the other changes between 1.5 libtiff and 1.6 libtiff were rather trivial...

[warmerdam]

The problem was a change in TIFFFlush() earlier this year intended to only rewrite the tile/strip offset/size maps if the only changes in the directory was related to them. The problem was that this did not clear the TIFF_BEENWRITING flag which confused things when the next directory was being written.

The fix has been applied upstream in libtiff/tif_flush.c, and GDAL's copy of libtiff has been refreshed (r15646).

[Even Rouault]

Test added in r15654. A small dataset with a flushcache() after filling and before building the overview was enough to trigger the bug.

Thanks for the fix.