Opened 9 years ago

Closed 9 years ago

#3673 closed defect (fixed)

GEORASTER - NODATA ELEMENT IN WRONG PLACE - ORA 13454

Reported by: dgeringe Owned by: ilucena
Priority: normal Milestone:
Component: GDAL_Raster Version: 1.7.2
Severity: normal Keywords: georaster
Cc: ilucena, warmerdam

Description

When loading a netcdf file into GeoRaster, validation fails with ORA 13454.

The cause is the NODATA element in the raster metadata is in the wrong place.

If it is manually moved right after the cellDepth element, the raster validates.

Change History (11)

comment:1 Changed 9 years ago by ilucena

Owner: changed from warmerdam to ilucena

This problem has passed unnoticeable because the data is still consistent and can be read by GDAL and even Oracle's GeoRasterViewer?.

Here is the section of the GeoRaster metadata, where the "rasterInfo" is showing the NODATA entry at the end:

<rasterInfo>
    <cellRepresentation>UNDEFINED</cellRepresentation>
    <cellDepth>16BIT_S</cellDepth>
    <totalDimensions>2</totalDimensions>
    <dimensionSize type="ROW">
      <size>512</size>
    </dimensionSize>
    <dimensionSize type="COLUMN">
      <size>512</size>
    </dimensionSize>
    <ULTCoordinate>
      <row>0</row>
      <column>0</column>
    </ULTCoordinate>
    <blocking>
      <type>REGULAR</type>
      <totalRowBlocks>2</totalRowBlocks>
      <totalColumnBlocks>2</totalColumnBlocks>
      <rowBlockSize>256</rowBlockSize>
      <columnBlockSize>256</columnBlockSize>
    </blocking>
    <interleaving>BSQ</interleaving>
    <pyramid>
      <type>NONE</type>
    </pyramid>
    <compression>
      <type>NONE</type>
    </compression>
    '''<NODATA>-9999</NODATA>'''
  </rasterInfo>

And the problem shows when running the function validateGeoRaster() from SDO_GEOR

Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL>
SQL> select sdo_geor.validateGeoraster(raster) from gdal_import t where t.raster.rasterid = 161;

SDO_GEOR.VALIDATEGEORASTER(RASTER)
--------------------------------------------------------------------------------
13454

Frank,

I don't think the GDAL's MiniXML allow us to force the order of new XML Elements under a certain node. Does it? Can you see how can we do that?

comment:2 Changed 9 years ago by ilucena

That is just the tip of the iceberg. The order shouldn't be a matter but it is.

The real problem is that GeoRaster supports a much more complex form of nodata notation than GDAL.

In GeoRaster a nodata notation could be just a value, a value range or an array of value ranges.

Examples of sdo_geor.addNodata() command on a 3 band georaster:

DECLARE
  gr sdo_georaster;
BEGIN
  SELECT raster INTO gr FROM gdal_import t WHERE t.raster.rasterid=151 FOR UPDATE;
  SDO_GEOR.addNODATA(gr, 0, sdo_range_array(sdo_range(0,10)));
  SDO_GEOR.addNODATA(gr, 1, sdo_range_array(sdo_range(2,12)));
  SDO_GEOR.addNODATA(gr, 2, sdo_range_array(sdo_range(1,11),sdo_range(21,31)));
  UPDATE gdal_import t SET raster=gr WHERE t.raster.rasterid=151;
END;

That creates one notadata entry for each band and for each band there is a range of pixel values supposed to be treated as notadata. Band number 3 actually contains 2 value ranges.

  <layerInfo>
    <layerDimension>BAND</layerDimension>
    <objectLayer>
      <layerNumber>0</layerNumber>
      <layerDimensionOrdinate>0</layerDimensionOrdinate>
      <layerID />
      <NODATA>
        <range>
          <min>0</min>
          <max>10</max>
        </range>
      </NODATA>
    </objectLayer>
    <subLayer>
      <layerNumber>1</layerNumber>
      <layerDimensionOrdinate>0</layerDimensionOrdinate>
      <layerID />
      <NODATA>
        <range>
          <min>2</min>
          <max>12</max>
        </range>
      </NODATA>
    </subLayer>
    <subLayer>
      <layerNumber>2</layerNumber>
      <layerDimensionOrdinate>1</layerDimensionOrdinate>
      <layerID />
      <NODATA>
        <range>
          <min>1</min>
          <max>11</max>
        </range>
        <range>
          <min>21</min>
          <max>31</max>
        </range>
      </NODATA>
    </subLayer>
  </layerInfo>

That doesn't seems to be possible to support but it should be possible to fix the support for just one nodata value per band.

comment:3 Changed 9 years ago by warmerdam

Cc: warmerdam added
Keywords: georaster added

Ivan,

The GDAL data model only permits a single nodata value for a GDALRasterBand. So if you need to support multiple values or more complex combinations for georaster it will need to be addressed via special creation options and/or driver specific metadata.

Another possibility is to implement support for band masks for georaster bands. On read you could interpret the complex nodata rules, and return a mask indicating invalid pixels. Details on band masks are available at:

http://trac.osgeo.org/gdal/wiki/rfc15_nodatabitmask

I will warn that relatively few applications currently utilize the masking capability, but it would be a step forward.

comment:4 Changed 9 years ago by ilucena

So in case we run this sequence:

% gdal_translate -of georaster input.fmt geor:usr/pwd@db,rdt,rid -co "NODATA=[50,105],[200,250]"
% gdal_translate -of gtiff geor:usr/pwd@db,rdt,rid output.tif

The output will preserve the original values plus a mask band with the combined notada(s). Right?

comment:5 Changed 9 years ago by warmerdam

Ivan,

I think the mask is preserved in geotiff by gdal_translate, but I am not certain.

comment:6 Changed 9 years ago by ilucena

I added support for mask band in GeoRaster about a year ago. So exporting from GeoRaster (with a real mask band) to a output driver that support mask band should work.

Bu the idea here is that if the GeoRaster driver finds any complex Nodata notation then it will lie saying "This dataset contain mask band!". And when gdal_translate is converting from GeoRaster to another format that support mask band, the GeoRaster driver will build the binary mask on the fly, in memory, in a ReadBlock? operation. If understand it correctly.

It sounds like a interesting solution but if not a lot of other output formats support mask band and since most applications don't care about it too, I am thinking that it will be better just a apply the mask while reading the regular data blocks.

comment:7 Changed 9 years ago by ilucena

The changes on revision r20012 will take care of the simple case, one nodata value for a single band GeoRaster.

comment:8 Changed 9 years ago by ilucena

The changes on revision r20130 will add support for per-band NoData? Value.

Note that even tough the GDAL API supports a single NoData? value per band, not all raster formats support that. So, if a GeoRaster contains one NoData? per band and it is converted to Geotiff, the Geotiff driver will use the first band Nodata as the dataset NoData?. The HFA driver and the Erdas Imagine Image format will handle that more precisely. Example:

% gdalinfo geor:scott/tiger,,gdal_import,raster,t.raster.rasterid=177 -mdd oracle
Driver: GeoRaster/Oracle Spatial GeoRaster
Files: none associated
Size is 4304, 3788
Coordinate System is:
PROJCS["WGS 84 / UTM zone 50N",
    GEOGCS["WGS 84",
        DATUM["WGS_1984",
            SPHEROID["WGS 84",6378137,298.257223563,
                AUTHORITY["EPSG","7030"]],
            AUTHORITY["EPSG","6326"]],
        PRIMEM["Greenwich",0,
            AUTHORITY["EPSG","8901"]],
        UNIT["degree",0.01745329251994328,
            AUTHORITY["EPSG","9122"]],
        AUTHORITY["EPSG","4326"]],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    PROJECTION["Transverse_Mercator"],
    PARAMETER["latitude_of_origin",0],
    PARAMETER["central_meridian",117],
    PARAMETER["scale_factor",0.9996],
    PARAMETER["false_easting",500000],
    PARAMETER["false_northing",0],
    AUTHORITY["EPSG","32650"],
    AXIS["Easting",EAST],
    AXIS["Northing",NORTH]]
Origin = (279036.375000000000000,-2288286.375000000000000)
Pixel Size = (57.000000000000000,-57.000000000000000)
Metadata (oracle):
  TABLE_NAME=scott.gdal_import
  COLUMN_NAME=raster
  RDT_TABLE_NAME=GDAL_RDT
  RASTER_ID=177
  SRID=32650
  WKT=PROJCS["WGS 84 / UTM zone 50N", GEOGCS [ "WGS 84", DATUM ["World Geodetic System 1984 (EPSG ID 6326)", SPHEROID ["WGS 84 (EPSG ID 7030)", 6378137.0, 298.257223563]], PRIMEM [ "Greenwich", 0.000000 ], UNIT ["Decimal Degree", 0.0174532925199433]], PROJECTION ["Transverse Mercator"], PARAMETER ["Latitude_Of_Origin", 0.0], PARAMETER ["Central_Meridian", 117.0], PARAMETER ["Scale_Factor", 0.9996], PARAMETER ["False_Easting", 500000.0], PARAMETER ["False_Northing", 0.0], UNIT ["Meter", 1.0]]
  METADATA=<georasterMetadata xmlns="http://xmlns.oracle.com/spatial/georaster">
  <objectInfo>
    <rasterType>21001</rasterType>
    <isBlank>false</isBlank>
    <defaultRed>1</defaultRed>
    <defaultGreen>1</defaultGreen>
    <defaultBlue>1</defaultBlue>
  </objectInfo>
  <rasterInfo>
    <cellRepresentation>UNDEFINED</cellRepresentation>
    <cellDepth>8BIT_U</cellDepth>
    <totalDimensions>3</totalDimensions>
    <dimensionSize type="ROW">
      <size>3788</size>
    </dimensionSize>
    <dimensionSize type="COLUMN">
      <size>4304</size>
    </dimensionSize>
    <dimensionSize type="BAND">
      <size>2</size>
    </dimensionSize>
    <ULTCoordinate>
      <row>0</row>
      <column>0</column>
      <band>0</band>
    </ULTCoordinate>
    <blocking>
      <type>REGULAR</type>
      <totalRowBlocks>15</totalRowBlocks>
      <totalColumnBlocks>17</totalColumnBlocks>
      <totalBandBlocks>2</totalBandBlocks>
      <rowBlockSize>256</rowBlockSize>
      <columnBlockSize>256</columnBlockSize>
      <bandBlockSize>1</bandBlockSize>
    </blocking>
    <interleaving>BSQ</interleaving>
    <pyramid>
      <type>NONE</type>
    </pyramid>
    <compression>
      <type>NONE</type>
    </compression>
  </rasterInfo>
  <spatialReferenceInfo>
    <isReferenced>true</isReferenced>
    <isRectified>true</isRectified>
    <SRID>32650</SRID>
    <spatialResolution dimensionType="X">
      <resolution>57</resolution>
    </spatialResolution>
    <spatialResolution dimensionType="Y">
      <resolution>57</resolution>
    </spatialResolution>
    <modelCoordinateLocation>CENTER</modelCoordinateLocation>
    <modelType>FunctionalFitting</modelType>
    <polynomialModel rowOff="0" columnOff="0" xOff="0" yOff="0" zOff="0" rowScale="1" columnScale="1" xScale="1" yScale="1" zScale="1">
      <pPolynomial pType="1" nVars="2" order="1" nCoefficients="3">
        <polynomialCoefficients>-40145.375 0 -0.01754385964912281</polynomialCoefficients>
      </pPolynomial>
      <qPolynomial pType="1" nVars="0" order="0" nCoefficients="1">
        <polynomialCoefficients>1</polynomialCoefficients>
      </qPolynomial>
      <rPolynomial pType="1" nVars="2" order="1" nCoefficients="3">
        <polynomialCoefficients>-4895.375 0.01754385964912281 0</polynomialCoefficients>
      </rPolynomial>
      <sPolynomial pType="1" nVars="0" order="0" nCoefficients="1">
        <polynomialCoefficients>1</polynomialCoefficients>
      </sPolynomial>
    </polynomialModel>
  </spatialReferenceInfo>
  <layerInfo>
    <layerDimension>BAND</layerDimension>
    <subLayer>
      <layerNumber>1</layerNumber>
      <layerDimensionOrdinate>0</layerDimensionOrdinate>
      <layerID />
      <NODATA>
        <value>10</value>
      </NODATA>
    </subLayer>
    <subLayer>
      <layerNumber>2</layerNumber>
      <layerDimensionOrdinate>1</layerDimensionOrdinate>
      <layerID />
      <NODATA>
        <value>20</value>
      </NODATA>
    </subLayer>
  </layerInfo>
</georasterMetadata>

Image Structure Metadata:
  INTERLEAVE=BAND
  COMPRESSION=NONE
Corner Coordinates:
Upper Left  (  279036.375,-2288286.375) (114d52'43.45"E, 20d40'51.52"S)
Lower Left  (  279036.375,-2504202.375) (114d51'0.13"E, 22d37'49.46"S)
Upper Right (  524364.375,-2288286.375) (117d14'2.25"E, 20d41'37.96"S)
Lower Right (  524364.375,-2504202.375) (117d14'13.66"E, 22d38'40.72"S)
Center      (  401700.375,-2396244.375) (116d 2'59.68"E, 21d40'0.36"S)
Band 1 Block=256x256 Type=Byte, ColorInterp=Gray
  NoData Value=10
Band 2 Block=256x256 Type=Byte, ColorInterp=Gray
  NoData Value=20

Converted to Geotiff:

% gdal_translate -of gtiff geor:scott/tiger,,gdal_import,raster,t.raster.rasterid=177 out.tif
Input file size is 4304, 3788
0...10...20...30...40...50...60...70...80...90...100 - done.

% gdalinfo out.tif
Driver: GTiff/GeoTIFF
Files: out.tif
       out.tif.aux.xml
Size is 4304, 3788
Coordinate System is:
PROJCS["WGS 84 / UTM zone 50N",
    GEOGCS["WGS 84",
        DATUM["WGS_1984",
            SPHEROID["WGS 84",6378137,298.257223563,
                AUTHORITY["EPSG","7030"]],
            AUTHORITY["EPSG","6326"]],
        PRIMEM["Greenwich",0],
        UNIT["degree",0.0174532925199433],
        AUTHORITY["EPSG","4326"]],
    PROJECTION["Transverse_Mercator"],
    PARAMETER["latitude_of_origin",0],
    PARAMETER["central_meridian",117],
    PARAMETER["scale_factor",0.9996],
    PARAMETER["false_easting",500000],
    PARAMETER["false_northing",0],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    AUTHORITY["EPSG","32650"]]
Origin = (279036.375000000000000,-2288286.375000000000000)
Pixel Size = (57.000000000000000,-57.000000000000000)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  (  279036.375,-2288286.375) (114d52'43.45"E, 20d40'51.52"S)
Lower Left  (  279036.375,-2504202.375) (114d51'0.13"E, 22d37'49.46"S)
Upper Right (  524364.375,-2288286.375) (117d14'2.25"E, 20d41'37.96"S)
Lower Right (  524364.375,-2504202.375) (117d14'13.66"E, 22d38'40.72"S)
Center      (  401700.375,-2396244.375) (116d 2'59.68"E, 21d40'0.36"S)
Band 1 Block=4304x1 Type=Byte, ColorInterp=Gray
  NoData Value=10
Band 2 Block=4304x1 Type=Byte, ColorInterp=Undefined
  NoData Value=10

Converted to Erdas Imagine Image:

% rm out.*
% gdal_translate -of hfa geor:scott/tiger,,gdal_import,raster,t.raster.rasterid=177 out.img
% gdalinfo out.img
Driver: HFA/Erdas Imagine Images (.img)
Files: out.img
       out.img.aux.xml
Size is 4304, 3788
Coordinate System is:
PROJCS["WGS_1984_UTM_Zone_50N",
    GEOGCS["GCS_WGS_1984",
        DATUM["WGS_1984",
            SPHEROID["WGS_1984",6378137,298.257223563]],
        PRIMEM["Greenwich",0],
        UNIT["Degree",0.017453292519943295]],
    PROJECTION["Transverse_Mercator"],
    PARAMETER["latitude_of_origin",0],
    PARAMETER["central_meridian",117],
    PARAMETER["scale_factor",0.9996],
    PARAMETER["false_easting",500000],
    PARAMETER["false_northing",0],
    UNIT["Meter",1]]
Origin = (279036.375000000000000,-2288286.375000000000000)
Pixel Size = (57.000000000000000,-57.000000000000000)
Corner Coordinates:
Upper Left  (  279036.375,-2288286.375) (114d52'43.45"E, 20d40'51.52"S)
Lower Left  (  279036.375,-2504202.375) (114d51'0.13"E, 22d37'49.46"S)
Upper Right (  524364.375,-2288286.375) (117d14'2.25"E, 20d41'37.96"S)
Lower Right (  524364.375,-2504202.375) (117d14'13.66"E, 22d38'40.72"S)
Center      (  401700.375,-2396244.375) (116d 2'59.68"E, 21d40'0.36"S)
Band 1 Block=64x64 Type=Byte, ColorInterp=Undefined
  Description = Layer_1
  NoData Value=10
  Metadata:
    LAYER_TYPE=athematic
Band 2 Block=64x64 Type=Byte, ColorInterp=Undefined
  Description = Layer_2
  NoData Value=20
  Metadata:
    LAYER_TYPE=athematic

I don't know if we should consider it a bug or a feature of Geotiff driver but it is a "users be aware" issue.

comment:9 Changed 9 years ago by ilucena

On r20266 I implemented full support for NODATA Values and Value Ranges.

More information about that GeoRaster's concept is available here:

http://download.oracle.com/docs/cd/E11882_01/appdev.112/e11827/geor_intro.htm#BABJCCBA

The way it is working now is that if there are just one regular NoData? values on a GeoRaster Layer then this value will be treated as a regular GDALRasterBand NoData? value.

If there are more than one value or a combination of values and value ranges then the drivel will act as if there was no NoData? value unless the Configuration Option or environment variable GEOR_FILTER_NODATA_VALUES exist.

If the GEOR_FILTER_NODATA_VALUES is present and set to any value other than "NO" then the driver will apply the filtering process as it is described by Oracle's documentation. It will look into all the pixel from each Band and it will compare against the 'Object Layer's NoData? and 'SubLayer?'s NoData?, if it is the case then it will replace the pixel by the first NoData? value on the list.

I ran a test with QGIS loading a GeoRaster with NoData? Values and Value Ranges and the results are looking pretty good.

Next step if to add that to the documentation.

But first, please let me know if that solution works for you.

comment:10 Changed 9 years ago by ilucena

Oops, I made a mistake on my last comment.

Correcting me:

If there are several NoData? values and/or NoData? value ranges the driver will assume by default that it suppose to do the filtering, replacing values as it was explained.

In that case, the only thing that will prevent the filtering process is the environment variable GEOR_FILTER_NODATA_VALUES set to NO. That could also be set at GDAL command lines utilities with "--configure GEOR_FILTER_NODATA_VALUES=NO" or programatically.

The logic behind it is that a visualization tool like QGIS and Mapserver will represent the image as expected by the user without any change in the software configuration.

comment:11 Changed 9 years ago by ilucena

Resolution: fixed
Status: newclosed
Note: See TracTickets for help on using tickets.