[PATCH - libtiff] TIFFMergeFieldInfo error when adding overviews to large tiff

[EliL]

This ticket comes from the email ​list

gdaladdo topo_mosaic_4326.tif 2 4 8 16 32 64 128 256 512 1024 --config COMPRESS_OVERVIEW PACKBITS ERROR 1: topo_mosaic_4326.tif:Failed to allocate memory for for fields array (1410046 elements of 16 bytes each)

gdalinfo -checksum topo_mosaic_4326.tif

Size is 398449, 210532

Even reported: "I've investigated a bit and I've discovered that while computing overviews, libtiff constantly switches from TIFF "directories" (main image, first overview,second overview, etc...). While doing this, it calls each time client code to install its own TIFF tags (libgeotiff for the geotiff tags, GDAL for the GDAL specific TIFF tags), but doesn't reset its array of existing tags. So each time the tags are added again and again, hence the huge memory allocation."

This can be observed on 32-bit windows with a large file like: Size is 398449, 210532, or on linux if built with additional instrumented components:

$ gdaladdo topo_mosaic_4326.tif 2 4 8 16 32 64 128 256 512 1024 --config 
COMPRESS_OVERVIEW PACKBITS
0...10...20...30...40...50...60...70...80...90...100 - done.
Current VSI memory usage        : 0 bytes
Maximum VSI memory usage        : 2361761086 bytes
Number of calls to VSIMalloc()  : 43981823
Number of calls to VSICalloc()  : 1259
Number of calls to VSIRealloc() : 17378556
Number of calls to VSIFree()    : 43983082
VSIMalloc + VSICalloc - VSIFree : 0

[Even Rouault]

Frank,

could you review this libtiff patch and commit it if you find it appropriate ? Another option would be to make TIFFMergeFieldInfo() more clever to detect duplicated in the entries that we insert and the existing entries, but that would be more complicated to code.

[Even Rouault]

upstreamed in libtiff CVS.

trunk r28236 "Internal libtiff: in TIFFDefaultDirectory(), reset any already existing extented tags installed by user code through the extender mechaninm before calling the extender callback (#5054)"

[Even Rouault]

Milestone renamed