Big TIFF is not producing a proper dataset with some files

[gaopeng]

There seems to two issues:

1) cannot create a big tiff when the default LZW compression is used because the output will be all 0/Nodata/255 2) cannot create a big tiff when a large dataset is used since it will produce rectangular pixels.

Test data will be sent in DVD.

[warmerdam]

I have confirmed problem (1). It turns out the code only flips into bigtiff producing mode automatically if there is no compression in play. This is apparently to avoid producing a bigtiff file (incompatible with most applications) in situations where a regular tiff would have done.

The work around is to pass the BIGTIFF=YES flag in any case where it is desirable to produce a bigtiff file. Testing to confirm that this works. I'm not sure what we could do that would be smarter than this as it is hard to switch things to be bigtiff once we have started writing.

[warmerdam]

By the way, I do not understand issue (2). Could you please elaborate on it?

[warmerdam]

I have confirmed the LZW translation works fine if BIGTIFF is forced via creation options.

[warmerdam]

Frank,

Thanks.

I think it's better for tiff driver to switch based on, e.g. uncompressed size. If I do it outside, I don't have any more info than that. Plus, The config key is global, and I will have set and restore. If that's too much, we can have another config key that means switch based on uncompressed size.

Gao

[warmerdam]

Gao,

Are you suggesting we switch to bigtiff for compressed files if there is any chance that the result will be larger than 4GB? I could do that but I'm concerned it will result in production of bigtiff files that are not compatible with much other software in cases where there is no need for the file to be a bigtiff.

I'm really not comfortable with that. My rule of thumb on the bigtiff conversion has been "first do no harm". That is, don't break something that was working fine before.

I could potentially have two new values for the BIGTIFF creation option:

1) IFNEEDED: gives the current behavior - only switch to BIGTIFF if it is certainly required. 2) IFSAFER: gives the behavior you want - essentially use bigtiff if there is any risk that it would be needed. In this case I'd likely switch to bigtiff if the uncompressed data is larger than 2GB as this also ensures we have room for internal overviews.

The default setting would be IFNEEDED (as now), and you could just set it to always be IFSAFER. Does that sound like a plan?

[warmerdam]

Frank,

Yes. That's a good plan.

Thanks, Gao

[warmerdam]

I have committed this changes in trunk (r15762). The actual options are BIGTIFF=IF_NEEDED and BIGTIFF=IF_SAFER.

I'm not planning to retrofit this into 1.5 at this time.