Opened 4 years ago

Closed 19 months ago

#6619 closed defect (wontfix)

Trying to build single, high, compressed overview level fails to allocate memory

Reported by: kthujvu Owned by: warmerdam
Priority: normal Milestone: closed_because_of_github_migration
Component: GDAL_Raster Version: svn-trunk
Severity: normal Keywords: gdaladdo, overviews, compression, memory
Cc:

Description

Trying to make gdaladdo to build single (just one level), high (level 512 and upwards), compressed (DEFLATE and LZW, didn't test others) overview level fails with

ERROR 2: overview.cpp, 3135: cannot allocate 16442459064 bytes

for level 512,

ERROR 2: overview.cpp, 3135: cannot allocate 60398960644 bytes

for level 1024,

ERROR 2: overview.cpp, 3135: cannot allocate 226494320644 bytes

for level 2048.

For example:

$ rm 50N000E_20101117_gmted_std075.tif.ovr
$ gdaladdo --config COMPRESS_OVERVIEW DEFLATE -ro 50N000E_20101117_gmted_std075.tif 1024
ERROR 2: overview.cpp, 3135: cannot allocate 60398960644 bytes
0...10...20...30...40...50...60...70...80...90...100 - done.
Overview building failed.

$ rm 50N000E_20101117_gmted_std075.tif.ovr
$ gdaladdo --config COMPRESS_OVERVIEW DEFLATE -ro 50N000E_20101117_gmted_std075.tif 512
ERROR 2: overview.cpp, 3135: cannot allocate 16442459064 bytes
0...10...20...30...40...50...60...70...80...90...100 - done.
Overview building failed.

$ rm 50N000E_20101117_gmted_std075.tif.ovr
$ gdaladdo --config COMPRESS_OVERVIEW DEFLATE -ro 50N000E_20101117_gmted_std075.tif 256
0...10...20...30...40...50...60...70...80...90...100 - done.
$ gdaladdo --config COMPRESS_OVERVIEW DEFLATE -ro 50N000E_20101117_gmted_std075.tif 512
ERROR 2: overview.cpp, 3135: cannot allocate 16442459064 bytes
0...10...20...30...40...50...60...70...80...90...100 - done.
Overview building failed.

If I specify multiple levels and start with a "working" one, it works perfectly fine:

$ rm 50N000E_20101117_gmted_std075.tif.ovr
$ gdaladdo --config COMPRESS_OVERVIEW DEFLATE -ro 50N000E_20101117_gmted_std075.tif 256 512 1024
0...10...20...30...40...50...60...70...80...90...100 - done.

This also happens for LZW with the same attempted byte allocation count, I did not test other compression algorithms.

If I do not use compression, it works perfectly fine:

$ gdaladdo -ro 50N000E_20101117_gmted_std075.tif 2048
0...10...20...30...40...50...60...70...80...90...100 - done.

Here is the sample that I used to reproduce this: ​https://transfer.sh/6IZid/50n000e-20101117-gmted-std075.tif.xz (~30MB, gone in 2 weeks!). If it is gone, just message me to re-upload.

For another, huge file the same errors happen with very similar, but not identical byte counts.

For a tiny file it works perfectly fine even with compression:

$ wget -nv https://svn.osgeo.org/gdal/trunk/autotest/gdrivers/data/small_world.tif
2016-08-10 21:39:34 URL:https://svn.osgeo.org/gdal/trunk/autotest/gdrivers/data/small_world.tif [240574/240574] -> "small_world.tif.1" [1]
$ gdaladdo --config COMPRESS_OVERVIEW DEFLATE -ro small_world.tif 2048
0...10...20...30...40...50...60...70...80...90...100 - done.

I have 8 gigabytes of RAM. Passing e.g. "--config GDAL_CACHEMAX 2048" does not change anything, I am not sure if it should though. gdal 35042 on Linux.

Change History (2)

comment:1 Changed 4 years ago by Even Rouault

This is an "expected" behaviour of the implementation. When downsampling with a huger overview factor, you need to fetch a lot of source pixels. When the overview is compressed, they are fetched in a single step so as to write the output block is one single time, which cause the error error. When the overview is not compressed, a different approach less memory intensive is taken.

This could perhaps be improved but require substantial rework.

So the workaround is to compute intermediate overview levels as you did.

comment:2 Changed 19 months ago by Even Rouault

Milestone: closed_because_of_github_migration
Resolution: wontfix
Status: newclosed

This ticket has been automatically closed because Trac is no longer used for GDAL bug tracking, since the project has migrated to GitHub?. If you believe this ticket is still valid, you may file it to https://github.com/OSGeo/gdal/issues if it is not already reported there.

Note: See TracTickets for help on using tickets.