Opened 16 years ago

Closed 16 years ago

#1893 closed defect (fixed)

Out of memory when reading large interlaced PNG

Reported by: warmerdam Owned by: warmerdam
Priority: normal Milestone:
Component: GDAL_Raster Version: unspecified
Severity: normal Keywords: png
Cc: gaopeng

Description

Frank, GDAL PNG driver can't handle the images described below. These images are interlaced, and large. The driver runs out memory when trying to read the entire image at once. Any ideas on how this can be fixed? I have sample images if you want.

Attachments (1)

gdal_svn_trunk_workaround_1893.patch (3.2 KB ) - added by Even Rouault 16 years ago.

Download all attachments as: .zip

Change History (7)

comment:1 by Even Rouault, 16 years ago

It's generaly considered that using PNG interlaced image is a bad idea nowadays since we have large network bandwiths. I would recommand you to convert your PNG interlaced image into non-interlaced or into another image format such as Tiff.

There's no efficient way in term of combined speed and memory usage to get a pixel value from an interlaced image (ADAM7 interlacing). You have to read almost the whole file to get the exact value of the seventh pixel of the first line.

However, I'm enclosing a patch that is a very rough workaround that translates the PNG to a temporary GTiff (/tmp/tmp.tif) and requires only the memory for one line to be allocated. As a consequence, it's not very speed efficient since we must read the previously written line for each pass.

For a real fix, I'm wondering if it wouldn't be better to make a standalone utility (png2gdal) that would do the job of converting big interlaced PNGs to another GDAL format ('convert' from ImageMagick can translate interlaced PNG to GTiff but I've not enough big interlaced PNGs to check if it requires a lot of memory or not).

Eventually we could keep the current strategy of reading the whole file in memory if the image is small (not more than 64 MB of decompressed data for example) and exit cleanly otherwise. We could also improve the patch but I'm wondering if there's a real benefit of "inlining" it as there are several points to solve :

  • where to put the temporary file ?
  • is it acceptable that GDAL hangs for a long time before providing the first data ?

by Even Rouault, 16 years ago

comment:2 by Even Rouault, 16 years ago

I forgot to mention that you must define the environment variable PNG_TO_GTIFF to enable the workaround.

comment:3 by warmerdam, 16 years ago

Status: newassigned

Even,

Use of temporary files is not a very satisfactory solution in my opinion, since it is fragile.

I plan to try an approach with a large chunk size that is still of a finite size (perhaps 100MB), and make a full image pass to produce each such chunk.

comment:4 by warmerdam, 16 years ago

I have introduced a change in trunk (r12527) to load big interlaced png files in 100MB chunks (100MB of uncompressed imagery, not 100MB in the PNG itself).

I'd appreciate some testing of this on a large file before I try porting this back to 1.4-esri if that is convenient. If not, I'll go ahead and port it for testing.

comment:5 by warmerdam, 16 years ago

Back ported into 1.4-esri as r12529 for testing.

comment:6 by warmerdam, 16 years ago

Resolution: fixed
Status: assignedclosed
Note: See TracTickets for help on using tickets.