gdaladdo crashes on a 16-bit dataset when external overviews are requested to have JPEG compression

[Even Rouault]

2 different problems :

import gdal
import os
try:
    os.remove('test.tif.ovr')
except:
    pass
ds = gdal.GetDriverByName('GTiff').Create('test.tif', 1024, 1024, 1, gdal.GDT_Int16)
ds = None
gdal.SetConfigOption('COMPRESS_OVERVIEW', 'JPEG')
gdal.SetConfigOption('PHOTOMETRIC_OVERVIEW', 'YCBCR')
ds = gdal.Open('test.tif')
ds.BuildOverviews('NEAREST', overviewlist = [2, 4, 8, 16])
ds = None

causes following crash

ERROR 1: JPEGSetupEncode:BitsPerSample 16 not allowed for JPEG
ERROR 1: test.tif.ovr:No space to read TIFF directory
ERROR 1: TIFFReadDirectory:Failed to read directory at offset 2310
ERROR 7: Assertion `nBlockXOff == 0' failed
in file `geotiff.cpp', line 468

Abandon

This is due to SetDirectory() failing and return code not being checked in geotiff.cpp.

If running "gdaladdo -ro --config COMPRESS_OVERVIEW JPEG test.tif 2 4 8 16" afterwards, we get a crash because gt_overview.cpp doesn't check that the overview bands are not NULL (Directory 1 from test.tif.ovr produced previously is empty)

ERROR 1: test.tif.ovr:No space to read TIFF directory
ERROR 1: TIFFReadDirectory:Failed to read directory at offset 2310
ERROR 1: test.tif.ovr:No space to read TIFF directory
ERROR 1: TIFFReadDirectory:Failed to read directory at offset 2310
ERROR 1: test.tif.ovr:No space to read TIFF directory
ERROR 1: TIFFReadDirectory:Failed to read directory at offset 2310
==8131== Invalid read of size 4
==8131==    at 0x43CC3B4: GDALRasterBand::GetXSize() (gdalrasterband.cpp:2387)
==8131==    by 0x43D6854: GDALRegenerateOverviewsMultiBand(int, GDALRasterBand**, int, GDALRasterBand***, char const*, int (*)(double, char const*, void*), void*) (overview.cpp:1169)
==8131==    by 0x422D4C5: GTIFFBuildOverviews (gt_overview.cpp:563)
==8131==    by 0x43B54F9: GDALDefaultOverviews::BuildOverviews(char const*, char const*, int, int*, int, int*, int (*)(double, char const*, void*), void*) (gdaldefaultoverviews.cpp:432)
==8131==    by 0x43B3050: GDALDataset::IBuildOverviews(char const*, int, int*, int, int*, int (*)(double, char const*, void*), void*) (gdaldataset.cpp:1376)
==8131==    by 0x4212433: GTiffDataset::IBuildOverviews(char const*, int, int*, int, int*, int (*)(double, char const*, void*), void*) (geotiff.cpp:2752)
==8131==    by 0x43B2CF5: GDALDataset::BuildOverviews(char const*, int, int*, int, int*, int (*)(double, char const*, void*), void*) (gdaldataset.cpp:1325)
==8131==    by 0x43B2DA9: GDALBuildOverviews (gdaldataset.cpp:1353)
==8131==    by 0x8048D9A: main (gdaladdo.cpp:136)
==8131==  Address 0x1c is not stack'd, malloc'd or (recently) free'd

[Even Rouault]

Fixed in trunk (r17414) and branches/1.6 (r17415)