Opened 21 months ago

Closed 17 months ago

Last modified 17 months ago

#6252 closed defect (fixed)

georaster compression=deflate read error depending on endianness

Reported by: kjancke Owned by: ilucena
Priority: normal Milestone:
Component: default Version: 1.11.2
Severity: normal Keywords: georaster compression endianness
Cc:

Description

Georaster produced on Big Endian machine (Solaris on SPARC) by gdal_translate employing compression=DEFLATE is not readable on Little Endian architecture.

Result on Little Endian (Debian Wheezy on x86_64):

bm4102@LinDevMap60:~$ gdalinfo geor:user/pw@machine,SDE_RDT_3778,24 -checksum
Driver: GeoRaster/Oracle Spatial GeoRaster
Files: none associated
Size is 640, 560
Coordinate System is:
GEOGCS["WGS 84",
    DATUM["WGS_1984",
        SPHEROID["WGS 84",6378137,298.257223563,
            AUTHORITY["EPSG","7030"]],
        AUTHORITY["EPSG","6326"]],
    PRIMEM["Greenwich",0,
        AUTHORITY["EPSG","8901"]],
    UNIT["degree",0.0174532925199433,
        AUTHORITY["EPSG","9122"]],
    AUTHORITY["EPSG","4326"]]
Origin = (6.500000000000000,60.000000000000000)
Pixel Size = (0.012500000000000,-0.012500000000000)
Image Structure Metadata:
  COMPRESSION=DEFLATE
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  (   6.5000000,  60.0000000) (  6d30' 0.00"E, 60d 0' 0.00"N)
Lower Left  (   6.5000000,  53.0000000) (  6d30' 0.00"E, 53d 0' 0.00"N)
Upper Right (  14.5000000,  60.0000000) ( 14d30' 0.00"E, 60d 0' 0.00"N)
Lower Right (  14.5000000,  53.0000000) ( 14d30' 0.00"E, 53d 0' 0.00"N)
Center      (  10.5000000,  56.5000000) ( 10d30' 0.00"E, 56d30' 0.00"N)
Band 1 Block=128x128 Type=Float32, ColorInterp=Gray
ERROR 1: ZLib return code (-3)
ERROR 1: ZLib return code (-3)
...
ERROR 1: ZLib return code (-3)
ERROR 1: ZLib return code (-3)
  Checksum=53696
  NoData Value=0
  Overviews: 320x280, 160x140, 80x70, 40x35, 20x17
ERROR 1: ZLib return code (-3)
ERROR 1: ZLib return code (-3)
...
ERROR 1: ZLib return code (-3)
ERROR 1: ZLib return code (-3)
  Overviews checksum: 62777, 64639, 65229, 71, 65201
bm4102@LinDevMap60:~$

Result on Big Endian (Solaris on Sparc):

[bm4102@utsira:/home/bm4102/gdal-1.11.2] gdalinfo geor:user/pw@machine,SDE_RDT_3778,24 -checksum
Driver: GeoRaster/Oracle Spatial GeoRaster
Files: none associated
Size is 640, 560
Coordinate System is:
GEOGCS["WGS 84",
    DATUM["WGS_1984",
        SPHEROID["WGS 84",6378137,298.257223563,
            AUTHORITY["EPSG","7030"]],
        AUTHORITY["EPSG","6326"]],
    PRIMEM["Greenwich",0,
        AUTHORITY["EPSG","8901"]],
    UNIT["degree",0.0174532925199433,
        AUTHORITY["EPSG","9122"]],
    AUTHORITY["EPSG","4326"]]
Origin = (6.500000000000000,60.000000000000000)
Pixel Size = (0.012500000000000,-0.012500000000000)
Image Structure Metadata:
  COMPRESSION=DEFLATE
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  (   6.5000000,  60.0000000)
Lower Left  (   6.5000000,  53.0000000)
Upper Right (  14.5000000,  60.0000000)
Lower Right (  14.5000000,  53.0000000)
Center      (  10.5000000,  56.5000000)
Band 1 Block=128x128 Type=Float32, ColorInterp=Gray
  Checksum=1553
  NoData Value=0
  Overviews: 320x280, 160x140, 80x70, 40x35, 20x17
  Overviews checksum: 378, 87, 18, 4, 2
[bm4102@utsira:/home/bm4102/gdal-1.11.2]

Change History (8)

comment:1 Changed 21 months ago by Even Rouault

Owner: changed from warmerdam to ilucena

comment:2 Changed 21 months ago by ilucena

The Oracle server is running on the SPARC machine. Is that right?

Debugging that error would require quite a lot of effort to get access to a similar environment, so I would like to ask some questions first:

When you compiled GDAL and the georaster driver on the Intel machine...

  • Did you use the macro CPL_MSB, CPL_LSB or WORDS_BIGENDIAN ?
  • Did you use the internal zlib, by passing "--with-zlib=internal" on "./configure" ?

As you may now, in GeoRaster the data is stored as Big Endian, no matter what platform the server is.

The GDAL that is running on the Intel Linux should call GDALSwapWords before the decompression unless there is something wrong with the way it was build.

http://trac.osgeo.org/gdal/browser/trunk/gdal/frmts/georaster/georaster_wrapper.cpp#L2011

Regards.

comment:3 Changed 20 months ago by kjancke

The questions above brought me on the right track:

The Oracle server runs on an MSB machine (AIX on Power7 hardware), the config log looked alright, zlib is taken from the system , none of CPL_MSB, CPL_LSB nor WORDS_BIGENDIAN was explicitly set.

Setting --with-zlib=internal and printing the bytes streamed into the MACRO PULLBYTE (frmts/zlib/inflate.c) to stderr clearly showed that the call to GDALSwapWords is the culprit: the stream is worked bytewise and the swapping disturbs the sequence of the data!

Removing the GDALSwapWords sequence from georaster_wrapper.cpp unconditionally keeps the stream inflatable and produces the correct result.

Regards Kai

Last edited 20 months ago by kjancke (previous) (diff)

comment:4 Changed 19 months ago by agodfrin

Like Ivan explained, we store all binary numbers in raster blobs in Oracle in BIG ENDIAN always - irrespective of the nature of the platform it runs on (BIG or LITTLE ENDIAN).

This means that a GDAL build running on a LITTLE ENDIAN (=essentially Intel chips: Linux on intel, Windows, Solaris on Intel) must swap byte order. This is what the code does:

#ifndef CPL_MSB
        if( nCellSizeBits > 8 )
        {
            int nWordSize  = nCellSizeBits / 8;
            int nWordCount = nColumnBlockSize * nRowBlockSize * nBandBlockSize;
            GDALSwapWords( pabyBlockBuf, nWordSize, nWordCount, nWordSize );
        }
#endif

I.e unless CPL_MSB is defined, the georaster plugin will swap the bytes. That is the default behavior and is the way GDAL builds on common platforms (Linux, Windows) work.

To run GDAL on a BIG ENDIAN platform (= Solaris on Sparc, AIX on Power7, HP Tru64, Linux on Power ...) must not swap byte order. That is achieved by defining CPL_MSB for builds on those platforms. The presence of CPL_MSB will effectively skip the byte swapping in the code above.

comment:5 Changed 19 months ago by ilucena

Hi Kai,

Could you give us some details on how did you manage to build GDAL on AIX?

Which compiler and version did you used?

How does you ./configure command looks likes?

Did you have to make changes to port/cpl_port.h specific to _AIX?

If you want to write direct to me, my e-mail is on frtms/georaster source code .cpp/.h files header.

Thanks,

Ivan

comment:6 Changed 17 months ago by ilucena

Fixed on revision #33771

GDALSwapWords should be called after the buffer is uncompressed.

comment:7 Changed 17 months ago by ilucena

Resolution: fixed
Status: newclosed

Backported to 2.0

Please give it a try.

comment:8 Changed 17 months ago by Even Rouault

branches/2.0 r33774 "backport r33771 bug #6252"

branches/2.0 r33881 "GeoRaster: use EQUALN instead of STARTS_WITH_CI since the later is 2.1only (follow up of r33774, #6252)"

Note: See TracTickets for help on using tickets.