INGR driver runs out of memory

[mpd]

The INGR driver uses a single block for non-tiled RLE files. On a large, but highly compressed, example (like the attached 31417 x 15708) GDALRasterBlock::Internalize (called from GDALRasterBand::IRasterIO via GDALRasterBand::GetLockedBlockRef) cannot allocate enough RAM (about 470MB) to handle the block.

[mpd]

Large, non-tiled, INGR RLE file (one part of two-part zip)

[mpd]

Large, non-tiled, INGR RLE file (one part of two-part zip)

[warmerdam]

Ivan,

What do you think? Is this practical to fix? The approach I'd imagine would be to treat large "one tile" images as having tiles that are a reasonable size (horizontal slices likely) but I'm not sure how easy that would be for you to accomplish.

I'm not going to press you to resolve this solution unless the paying client also feels strongly about it.

[ilucena]

You are right, RLE can be decompressed in blocks. I think I can make the driver to set a block size and fill it up as asked instead of all at once. It is going to be a little bit of trouble for random access tough. The problem with that format is that it has been deprecated by Intergraph, they suggest the use of tiled CCITT instead.

[mpd]

I would not go as far as saying that I feel strongly about it, but there is a danger that I will eventually. Our old in-house INGR Plug-in handled this image OK, because we were able to support is as true 1-bit and therefore only needed 1/8th of the RAM. But it is just a random sample, not an end-user file.

If random access is required then could the driver read the whole of the 1-bit data into RAM (<60Mb for that image) and serve "virtual" blocks from that? If that were an easier solution than decompressing into blocks then I'd be happy with that.

[warmerdam]

This has come up with a client of mine as a pressing issue so I'm going to take a crack at it.

My thinking is to load the whole RLE compressed image into a buffer, but to treat things as a block per scanline and to keep track of offsets into the RLE buffer for each scanline.

[ilucena]

That looks very clever.

+1

[warmerdam]

I seem to have it working for paletted images.

Martin - could you email me your .rle file if you still have it? I'm having trouble extracting from your multifile zip dataset.

[warmerdam]

I have done a preliminary implementation (r15514) only for paletted RLE compressed images. I think the same approach should be easy to extend to bitonal and unpalleted images. I'd like to get or generate a few more test images to have greater confidence in the effort.

[warmerdam]

I have made additional changes to support reading bitonal RLE images one scanline at a time in trunk (r15527). I would note that the INGR_DecodeRunLengthBitonal() was substantially restructured and could benefit from additional testing and review. It does work fine with Martin's file.

There still appears to be an RLE type that maps to INGR_DeocodeRunLength() (I'm not sure of the corresponding format code) that is not handled on a scanline oriented basis.

Ivan - any testing you can do with this would be appreciated.

[ilucena]

I did some test and I noticed that the Tiled-RLE-Bitonal support was broken.

That was because the code was delete on (r15527). See, strange enough the format 9 follow two different logic, one that seams to be used only for tiled data blocks and other for untiled data blocks. See (1).

The untiled data blocks uses a special key 0x5900 to identify the beginning of a new line followed by that suppose to be line id and line size. Note that those two fields are usually neglected by data providers.

Tiled data blocks follow a more standard Bitonal RLE definition. More information on (2). I restored the code as it was before r15527 (1) but I found some problems. I fixed those problems and tested it with data sets generated by the Intergraph freeware "Image Station". The results are very good.

[1] http://trac.osgeo.org/gdal/browser/trunk/gdal/frmts/ingr/IngrTypes.cpp?rev=15573#L1058

[2] ​http://oreilly.com/www/centers/gff/formats/ingr/index.htm

[warmerdam]

Ivan,

I'm now seeing that the one block bitonal files are not working properly in scanline-at-a-time mode. For instance, the ingr_8 test suite test is failing, and Martin's test file translated in corrupt form.

Can we add a tiled bitonal file to the test suite?

I'm going to morph it so there are two versions of the bitonal decompress function - one used for tiled files and one for untiled files. Hopefully that will provide for all needs, though it would be cleaner if we could come up with a unified approach.

[warmerdam]

I have split the bitonal decoding into two functions - one for tiled and one for untiled (r15581). I'd appreciate your confirming that tiled still works.

If possible, I'd also appreciate adding a tiled rle compressed bitonal image to the ingr.py test suite to avoid future regression.

Thanks for all your work on this, and sorry for botching the tiled files.

[warmerdam]

It would be ideal to have a test for this but I don't think we need to keep the ticket open.

[Even Rouault]

Milestone 1.6.4 deleted