Can't open binary grids created with ArcGIS 10

[cdestigter]

We're having problems with a binary grid created in ArcGIS 10. The equivalent grid created in 9.3 works fine.

With reference to the ​AIG format notes, it seems a few things are different than gdal expects:

• magic number is 00 00 27 0A FF FF FC 08, (not one of the two known magic numbers listed)
• tile type for many tiles (seems to be the ones with NODATA as the top left pixel) is 0x80 (not one of the known tile types in gdal/frmts/aigrid )
• RMinSize is 0x0 for each tile (seems odd? maybe okay)

The statistics are also a bit weird for the arcgis 10 grid (they're all zero):

    Min=1.000 Max=6.000 
    Minimum=0.000, Maximum=0.000, Mean=0.000, StdDev=0.000
    NoData Value=255
    Overviews: 1459x1000, 730x500, 365x250
    Metadata:
      STATISTICS_MINIMUM=0
      STATISTICS_MAXIMUM=0
      STATISTICS_MEAN=0
      STATISTICS_STDDEV=0

Not sure why there are a Min/Max but also different min/max statistics in the metadata.

With client permission, I've uploaded the two grids for testing purposes, as well as a small python script that reproduces the problem. They're here: ​http://dl.dropbox.com/u/2649727/gdal-aig-test.zip

The test script shows that after converting to GTiff, the ArcGIS 10 grid has nothing but zeroes in it, but the 9.3 grid has the correct data (values 1-6).

Tested on gdal 1.7.3

[warmerdam]

I have confirmed that reading the ArcGIS 10 generated raster does not work with GDAL (including trunk). The error in the statistics seems to be due to the .aux.xml file, which was presumably generated with GDAL in ArcGIS 10 and so gets the wrong values when computing by reading the file.

I have observed that the ArcGIS 10 tiles are 256x16 instead of the usual 256x4 and that the ArcGIS 10 file is about 100 times larger than in 9.3. It seems 0x80 tiles are being widely used for nodata areas and they appear to be essentially uncompressed (and unreadable to GDAL). Also the 0x00 blocks are not being read properly for some reason.

Clearly some work to do!

I'm going to check with ESRI to see if they want to offer any suggestions.

[warmerdam]

It seems the difference is that ArcGIS 10 supports writing uncompressed integer grids which behave somewhat differently. I believe the "uncompressed" flag is at by 20 in the hdr.adf file and if "1" then blocks should be treated as uncompressed in a similar manner to float data.

I have applied the corresponding change in trunk (r22159) and 1.8 branch (r22160).

It would be nice to have a very small (less than 20K) file demonstrating uncompressed grid format that I could incorporate in the test suite.

[warmerdam]

I have briefly updated the format docs accordingly.