Ticket #1893 (closed defect: fixed)

Opened 1 year ago

Last modified 1 year ago

Out of memory when reading large interlaced PNG

Reported by: warmerdam Assigned to: warmerdam
Priority: normal Milestone:
Component: GDAL_Raster Version: unspecified
Severity: normal Keywords: png
Cc: gaopeng

Description

Frank, GDAL PNG driver can't handle the images described below. These images are interlaced, and large. The driver runs out memory when trying to read the entire image at once. Any ideas on how this can be fixed? I have sample images if you want.

Attachments

gdal_svn_trunk_workaround_1893.patch (3.2 kB) - added by rouault on 10/08/07 15:52:20.

Change History

10/08/07 15:51:38 changed by rouault

It's generaly considered that using PNG interlaced image is a bad idea nowadays since we have large network bandwiths. I would recommand you to convert your PNG interlaced image into non-interlaced or into another image format such as Tiff.

There's no efficient way in term of combined speed and memory usage to get a pixel value from an interlaced image (ADAM7 interlacing). You have to read almost the whole file to get the exact value of the seventh pixel of the first line.

However, I'm enclosing a patch that is a very rough workaround that translates the PNG to a temporary GTiff (/tmp/tmp.tif) and requires only the memory for one line to be allocated. As a consequence, it's not very speed efficient since we must read the previously written line for each pass.

For a real fix, I'm wondering if it wouldn't be better to make a standalone utility (png2gdal) that would do the job of converting big interlaced PNGs to another GDAL format ('convert' from ImageMagick? can translate interlaced PNG to GTiff but I've not enough big interlaced PNGs to check if it requires a lot of memory or not).

Eventually we could keep the current strategy of reading the whole file in memory if the image is small (not more than 64 MB of decompressed data for example) and exit cleanly otherwise. We could also improve the patch but I'm wondering if there's a real benefit of "inlining" it as there are several points to solve :

  • where to put the temporary file ?
  • is it acceptable that GDAL hangs for a long time before providing the first data ?

10/08/07 15:52:20 changed by rouault

  • attachment gdal_svn_trunk_workaround_1893.patch added.

10/08/07 15:53:24 changed by rouault

I forgot to mention that you must define the environment variable PNG_TO_GTIFF to enable the workaround.

10/24/07 15:57:06 changed by warmerdam

  • status changed from new to assigned.

Even,

Use of temporary files is not a very satisfactory solution in my opinion, since it is fragile.

I plan to try an approach with a large chunk size that is still of a finite size (perhaps 100MB), and make a full image pass to produce each such chunk.

10/24/07 20:13:14 changed by warmerdam

I have introduced a change in trunk (r12527) to load big interlaced png files in 100MB chunks (100MB of uncompressed imagery, not 100MB in the PNG itself).

I'd appreciate some testing of this on a large file before I try porting this back to 1.4-esri if that is convenient. If not, I'll go ahead and port it for testing.

10/24/07 20:37:28 changed by warmerdam

Back ported into 1.4-esri as r12529 for testing.

11/03/07 11:03:00 changed by warmerdam

  • status changed from assigned to closed.
  • resolution set to fixed.