stripes in ArcInfo binary grid

[bartvde]

art van den Eijnden (OSGIS) wrote:

Hi list,

after upgrading to GDAL 1.5 (coming from 1.4) we have some stripes in an ArcInfo binary grid (viewed with Mapserver).

Any ideas what could be causing this?

The problem can be seen in this screendump:

​http://www.osgis.nl/download/gdal15problem.png

Bart,

If you can file a ticket on this, and provide a pointer to the problem dataset I'll treat this as priority to fix before the soon to be released GDAL 1.5.1.

There have been some improvements to the tile decoder for arc/info grids for 1.5.0 but with no specifications and only a haphazard assortment of test data it is hard to be sure one improvement doesn't break some other case.

Best regards,

[bartvde]

Mapserver LAYER definition:

  LAYER
    CLASSITEM "[pixel]"
    DATA "ahn_100/geogegevens/raster/landsdekkend/ahn_100/w001001.adf"
    DUMP TRUE
    GROUP "AHN"
    MAXSCALE 5e+06
      METADATA
        "WCS_LABEL"     "Actueel Hoogtebestand van Nederland (AHN).Landsdekkende hoogteinformatie."
        "WMS_METADATAURL_TYPE"  "TC211"
        "WCS_SIZE"      "2700 3197"
        "WCS_BANDCOUNT" "1"
        "WCS_FORMATS"   "GEOTIFF,ECW,MRSID"
        "WCS_RESOLUTION"        "100 100"
        "OWS_ABSTRACT"  "AHN in de vorm van een grid met een resolutie van 100 meter waarbij de hoogtewaarde is opgenomen in centimeters. Het 25x25 en 100x100 meter DHM worden beide afgeleid uit het 5x5 meter DHM. De hoogtewaardes worden berekend door het gemiddelde te berekenen van de onderliggende 5x5 meter gridcellen. Doordat 'toevallige' fouten in de 5x5 meter gridcellen voor een groot deel zullen worden uitgemiddeld zal de nauwkeurigheid van het 25x25 meter DHM groter worden. Dit effect zal zelfs nog iets groter zijn bij het 100 x 100 meter DHM.In het algemeen geldt dat als de gridcelgrootte wordt vergroot, de nauwkeurigheid toeneemt. Echter, de gedetailleerdheid van de terreinbeschrijving neemt ook af bij grotere gridcellen. Dit effect manifesteert zich met name bij het optreden van scherpe overgangen in het terrein, zoals bijvoorbeeld bij een dijklichaam. In deze verbeterde versie van het AHN is de NAP-correctie doorgevoerd evenals enkele andere verbeteringen.Het AHN is een gezamenlijk initiatief van Rijkswaterstaat, provincies en waterschappen voor de bouw van een nieuw, landsdekkend, digitaal hoogtebestand. In 1996 is begonnen met de bouw van het AHN, dat sinds begin 2004 voor geheel Nederland beschikbaar is. Het is de bedoeling dat het iedere 5 tot 10 jaar wordt geactualiseerd."
        "WCS_RANGESET_LABEL"    "Hoogtes in meters"
        "WFS_MAXFEATURES"       "50"
        "OWS_EXTENT"    "10000 300000 280000 619700"
        "geoservices_iso19119_status"   "failed"
        "GML_EXCLUDE_ITEMS"     "class"
        "OWS_TITLE"     "Actueel Hoogtebestand Nederland 100 meter"
        "WMS_METADATAURL_FORMAT"        "text/xml"
        "GML_INCLUDE_ITEMS"     "all"
      END
    MINSCALE 250000
    NAME "AAA14"
    OFFSITE 255 255 255
    PROCESSING "SCALE=-945,33569"
    PROCESSING "SCALE_BUCKETS=2000"
    PROJECTION
      "init=epsg:28992"
    END
    SIZEUNITS PIXELS
    STATUS ON
    TEMPLATE "blank.html"
    TOLERANCEUNITS PIXELS
    TYPE RASTER
    UNITS METERS
    CLASS
      NAME "-1000 tot -500 cm"
      EXPRESSION ([pixel] > -1000 AND [pixel] < -500)
      METADATA
      END
      STYLE
        ANGLE 360
        COLOR 0 58 229
        SYMBOL 0
      END
    END
    CLASS
      NAME "-500 tot -300 cm"
      EXPRESSION ([pixel] >= -500 AND [pixel] < -300)
      METADATA
      END
      STYLE
        ANGLE 360
        COLOR 0 129 254
        SYMBOL 0
      END
    END
    CLASS
      NAME "-300 tot -200 cm"
      EXPRESSION ([pixel] >= -300 AND [pixel] < -200)
      METADATA
      END
      STYLE
        ANGLE 360
        COLOR 0 166 199
        SYMBOL 0
      END
    END
    CLASS
      NAME "-200 tot -100 cm"
      EXPRESSION ([pixel] >= -200 AND [pixel] < -100)
      METADATA
      END
      STYLE
        ANGLE 360
        COLOR 110 179 254
        SYMBOL 0
      END
    END
    CLASS
      NAME "-100 tot -50 cm"
      EXPRESSION ([pixel] >= -100 AND [pixel] < -50)
      METADATA
      END
      STYLE
        ANGLE 360
        COLOR 134 206 254
        SYMBOL 0
      END
    END
   CLASS
      NAME "-50 tot 0 cm"
      EXPRESSION ([pixel] >= -50 AND [pixel] < 0)
      METADATA
      END
      STYLE
        ANGLE 360
        COLOR 179 224 254
        SYMBOL 0
      END
    END
    CLASS
      NAME "0 tot 50 cm"
      EXPRESSION ([pixel] >= 0 AND [pixel] < 50)
      METADATA
      END
      STYLE
        ANGLE 360
        COLOR 34 169 128
        SYMBOL 0
      END
    END
    CLASS
      NAME "50 tot 100 cm"
      EXPRESSION ([pixel] >= 50 AND [pixel] < 100)
      METADATA
      END
      STYLE
        ANGLE 360
        COLOR 54 169 94
        SYMBOL 0
      END
    END
    CLASS
      NAME "100 tot 150 cm"
      EXPRESSION ([pixel] >= 100 AND [pixel] < 150)
      METADATA
      END
      STYLE
        ANGLE 360
        COLOR 97 254 96
        SYMBOL 0
      END
    END
    CLASS
      NAME "150 tot 200 cm"
      EXPRESSION ([pixel] >= 150 AND [pixel] < 200)
      METADATA
      END
      STYLE
        ANGLE 360
        COLOR 150 254 149
        SYMBOL 0
      END
    END
    CLASS
      NAME "200 tot 300 cm"
      EXPRESSION ([pixel] >= 200 AND [pixel] < 300)
      METADATA
      END
      STYLE
        ANGLE 360
        COLOR 127 255 212
        SYMBOL 0
      END
    END
    CLASS
      NAME "300 tot 400 cm"
      EXPRESSION ([pixel] >= 300 AND [pixel] < 400)
      METADATA
      END
      STYLE
        ANGLE 360
        COLOR 152 251 152
        SYMBOL 0
      END
    END
    CLASS
      NAME "400 tot 600 cm"
      EXPRESSION ([pixel] >= 400 AND [pixel] < 600)
      METADATA
      END
      STYLE
        ANGLE 360
        COLOR 165 182 85
        SYMBOL 0
      END
    END
    CLASS
      NAME "600 tot 1000 cm"
      EXPRESSION ([pixel] >= 600 AND [pixel] < 1000)
      METADATA
      END
      STYLE
        ANGLE 360
        COLOR 226 254 95
        SYMBOL 0
      END
    END
    CLASS
      NAME "1000 tot 1500 cm"
      EXPRESSION ([pixel] >= 1000 AND [pixel] < 1500)
      METADATA
      END
      STYLE
        ANGLE 360
        COLOR 254 248 164
        SYMBOL 0
      END
    END
    CLASS
      NAME "1500 tot 2000 cm"
      EXPRESSION ([pixel] >= 1500 AND [pixel] < 2000)
      METADATA
      END
      STYLE
        ANGLE 360
        COLOR 255 235 205
        SYMBOL 0
      END
    END
    CLASS
      NAME "2000 tot 3000 cm"
      EXPRESSION ([pixel] >= 2000 AND [pixel] < 3000)
      METADATA
      END
      STYLE
        ANGLE 360
        COLOR 254 203 159
        SYMBOL 0
      END
    END
    CLASS
      NAME "3000 tot 4000 cm"
      EXPRESSION ([pixel] >= 3000 AND [pixel] < 4000)
      METADATA
      END
      STYLE
        ANGLE 360
        COLOR 233 150 122
        SYMBOL 0
      END
    END
    CLASS
      NAME "4000 tot 6000 cm"
      EXPRESSION ([pixel] >= 4000 AND [pixel] < 6000)
      METADATA
      END
      STYLE
        ANGLE 360
        COLOR 205 92 92
        SYMBOL 0
      END
    END
    CLASS
      NAME "6000 tot 10000 cm"
      EXPRESSION ([pixel] >= 6000 AND [pixel] < 10000)
      METADATA
      END
      STYLE
        ANGLE 360
        COLOR 121 82 48
        SYMBOL 0
      END
    END
    CLASS
      NAME "10000 tot 15000 cm"
      EXPRESSION ([pixel] >= 10000 AND [pixel] < 15000)
      METADATA
      END
      STYLE
        ANGLE 360
        COLOR 111 40 40
        SYMBOL 0
      END
    END
    CLASS
      NAME "15000 tot 20000 cm"
      EXPRESSION ([pixel] >= 15000 AND [pixel] < 20000)
      METADATA
      END
      STYLE
        ANGLE 360
        COLOR 105 0 0
        SYMBOL 0
      END
    END
    CLASS
      NAME "20000 tot 25000 cm"
      EXPRESSION ([pixel] >= 20000 AND [pixel] < 25000)
      METADATA
      END
      STYLE
        ANGLE 360
        COLOR 169 0 0
        SYMBOL 0
      END
    END
    CLASS
      NAME "25000 tot 30000 cm"
      EXPRESSION ([pixel] >= 25000 AND [pixel] < 30000)
      METADATA
      END
      STYLE
        ANGLE 360
        COLOR 229 0 0
        SYMBOL 0
      END
    END
  END

[warmerdam]

It turns out the problem is actually the HFA driver reading the .rrd overviews, not the AIGRID driver. A temporary work around would be to delete the .aux and .rrd file and regenerate overviews with gdaladdo.

Digging deeper into the HFA problem.

[warmerdam]

I have reviewed a problem tile from the .rrd file. The tile is 256x4 and starts at offset 0x347B and is 145 bytes long (to 0x350C).

00003470  01 00 00 00 0f 00 00 00  00 44 00 00 00 00 00 80  |.........D......|
00003480  00 00 00 91 00 00 00 00  40 e1 01 01 01 01 01 01  |........@.......|
00003490  01 01 01 01 01 01 01 01  01 01 01 01 01 01 01 01  |................|
000034a0  01 01 01 01 01 01 01 01  01 40 e1 01 01 01 01 01  |.........@......|
000034b0  01 01 01 01 01 01 01 01  01 01 01 01 01 01 01 01  |................|
000034c0  01 01 01 01 01 01 01 01  01 01 40 e1 01 01 01 01  |..........@.....|
000034d0  01 01 01 01 01 01 01 01  01 01 01 01 01 01 01 01  |................|
000034e0  01 01 01 01 01 01 01 01  01 01 01 40 e1 01 01 01  |...........@....|
000034f0  01 01 01 01 01 01 01 01  01 01 01 01 01 01 01 01  |................|
00003500  01 01 01 01 01 01 01 01  01 01 01 01 05 00 00 00  |................|

The UncompressBlock() decodes this as nMinVal=0, nNumRuns=128, nDataOffset=145 and nNumBits=0. This decodes as 1024 pixels with value zero *but* clearly some of those pixels are intended to be the proper min-val for the dataset which is -2147483647 (-MIN_INT). I have been unable to deduce how this tiled can actually be decoded. The current code behavior seems correct for the data available even though it clearly doesn't produce the expected result.

I looked back in the code and I can't see any obvious way this could ever have worked. However, r11142 for #1000 likely changed the behavior from returning the nodata value for all pixels to returning zeros for all pixels. Perhaps it wasn't really working before, but it was less obvious?

Adding Sam to the ticket since he was involved in the earlier changes and might have some wisdom to offer.

Also pushing off to 1.5.2 and reducing priority since I'm not sure it is solvable, and it isn't clear this really represents a regression compared to 1.4.x.

[bartvde]

Frank, I can confirm the workaround works fine. Thanks.

[Sam Gillingham]

The more I look into these problems the less I think we understand the format completely. Unfortunately I don't seem to be getting anywhere with Leica in regards to the format specification (see #1000). I'll see if I can work out another angle on them.

[brunofriedmann]

Hi All, we are having typically the same trouble here. But we don't use Arcinfo type.

There's color stripes on rendered images. You will find the image issue from mapserver.

This appear on our server from the update of gdal from 1.4x to 1.5x (actually 1.5.1 rpm made by the openbuilder opensuse website). we are on 64bits servers.

​http://linux.ioda.net/images/n16_defect.jpg

This tend to prove that it's not probably a trouble attached to arcinfo.

[Jukka Rahkonen]

bartvde, do you believe that this issue is still possible to reproduce with reasonable pain? Or would it be better to close the ticket?

[bartvde]

this is so old, let's close it ;-)