Opened 8 years ago

Closed 7 years ago

Last modified 6 years ago

#5054 closed defect (fixed)

[PATCH - libtiff] TIFFMergeFieldInfo error when adding overviews to large tiff

Reported by: EliL Owned by: warmerdam
Priority: normal Milestone: 2.0.0
Component: GDAL_Raster Version: svn-trunk
Severity: normal Keywords: libtiff, tiff tags, overviews
Cc:

Description

This ticket comes from the email list

gdaladdo topo_mosaic_4326.tif 2 4 8 16 32 64 128 256 512 1024 --config COMPRESS_OVERVIEW PACKBITS ERROR 1: topo_mosaic_4326.tif:Failed to allocate memory for for fields array (1410046 elements of 16 bytes each)

gdalinfo -checksum topo_mosaic_4326.tif

Size is 398449, 210532

Even reported: "I've investigated a bit and I've discovered that while computing overviews, libtiff constantly switches from TIFF "directories" (main image, first overview,second overview, etc...). While doing this, it calls each time client code to install its own TIFF tags (libgeotiff for the geotiff tags, GDAL for the GDAL specific TIFF tags), but doesn't reset its array of existing tags. So each time the tags are added again and again, hence the huge memory allocation."

This can be observed on 32-bit windows with a large file like: Size is 398449, 210532, or on linux if built with additional instrumented components:

$ gdaladdo topo_mosaic_4326.tif 2 4 8 16 32 64 128 256 512 1024 --config 
COMPRESS_OVERVIEW PACKBITS
0...10...20...30...40...50...60...70...80...90...100 - done.
Current VSI memory usage        : 0 bytes
Maximum VSI memory usage        : 2361761086 bytes
Number of calls to VSIMalloc()  : 43981823
Number of calls to VSICalloc()  : 1259
Number of calls to VSIRealloc() : 17378556
Number of calls to VSIFree()    : 43983082
VSIMalloc + VSICalloc - VSIFree : 0

Attachments (1)

libtiff_fix_growing_customfields.patch (1004 bytes) - added by Even Rouault 8 years ago.

Download all attachments as: .zip

Change History (4)

Changed 8 years ago by Even Rouault

comment:1 Changed 8 years ago by Even Rouault

Summary: TIFFMergeFieldInfo error when adding overviews to large tiff[PATCH - libtiff] TIFFMergeFieldInfo error when adding overviews to large tiff

Frank,

could you review this libtiff patch and commit it if you find it appropriate ? Another option would be to make TIFFMergeFieldInfo() more clever to detect duplicated in the entries that we insert and the existing entries, but that would be more complicated to code.

comment:2 Changed 7 years ago by Even Rouault

Component: defaultGDAL_Raster
Milestone: 2.0
Resolution: fixed
Status: newclosed

upstreamed in libtiff CVS.

trunk r28236 "Internal libtiff: in TIFFDefaultDirectory(), reset any already existing extented tags installed by user code through the extender mechaninm before calling the extender callback (#5054)"

comment:3 Changed 6 years ago by Even Rouault

Milestone: 2.02.0.0

Milestone renamed

Note: See TracTickets for help on using tickets.