Out of memory when reading large interlaced PNG

[warmerdam]

Frank, GDAL PNG driver can't handle the images described below. These images are interlaced, and large. The driver runs out memory when trying to read the entire image at once. Any ideas on how this can be fixed? I have sample images if you want.

[rouault]

It's generaly considered that using PNG interlaced image is a bad idea nowadays since we have large network bandwiths. I would recommand you to convert your PNG interlaced image into non-interlaced or into another image format such as Tiff.

There's no efficient way in term of combined speed and memory usage to get a pixel value from an interlaced image (ADAM7 interlacing). You have to read almost the whole file to get the exact value of the seventh pixel of the first line.

However, I'm enclosing a patch that is a very rough workaround that translates the PNG to a temporary GTiff (/tmp/tmp.tif) and requires only the memory for one line to be allocated. As a consequence, it's not very speed efficient since we must read the previously written line for each pass.

For a real fix, I'm wondering if it wouldn't be better to make a standalone utility (png2gdal) that would do the job of converting big interlaced PNGs to another GDAL format ('convert' from ImageMagick? can translate interlaced PNG to GTiff but I've not enough big interlaced PNGs to check if it requires a lot of memory or not).

Eventually we could keep the current strategy of reading the whole file in memory if the image is small (not more than 64 MB of decompressed data for example) and exit cleanly otherwise. We could also improve the patch but I'm wondering if there's a real benefit of "inlining" it as there are several points to solve :

• where to put the temporary file ?
• is it acceptable that GDAL hangs for a long time before providing the first data ?

[rouault]

I forgot to mention that you must define the environment variable PNG_TO_GTIFF to enable the workaround.

[warmerdam]

Even,

Use of temporary files is not a very satisfactory solution in my opinion, since it is fragile.

I plan to try an approach with a large chunk size that is still of a finite size (perhaps 100MB), and make a full image pass to produce each such chunk.

[warmerdam]

I have introduced a change in trunk (r12527) to load big interlaced png files in 100MB chunks (100MB of uncompressed imagery, not 100MB in the PNG itself).

I'd appreciate some testing of this on a large file before I try porting this back to 1.4-esri if that is convenient. If not, I'll go ahead and port it for testing.

[warmerdam]

Back ported into 1.4-esri as r12529 for testing.