#6252 closed defect (fixed)
georaster compression=deflate read error depending on endianness
Reported by: | kjancke | Owned by: | ilucena |
---|---|---|---|
Priority: | normal | Milestone: | |
Component: | default | Version: | 1.11.2 |
Severity: | normal | Keywords: | georaster compression endianness |
Cc: |
Description
Georaster produced on Big Endian machine (Solaris on SPARC) by gdal_translate employing compression=DEFLATE is not readable on Little Endian architecture.
Result on Little Endian (Debian Wheezy on x86_64):
bm4102@LinDevMap60:~$ gdalinfo geor:user/pw@machine,SDE_RDT_3778,24 -checksum Driver: GeoRaster/Oracle Spatial GeoRaster Files: none associated Size is 640, 560 Coordinate System is: GEOGCS["WGS 84", DATUM["WGS_1984", SPHEROID["WGS 84",6378137,298.257223563, AUTHORITY["EPSG","7030"]], AUTHORITY["EPSG","6326"]], PRIMEM["Greenwich",0, AUTHORITY["EPSG","8901"]], UNIT["degree",0.0174532925199433, AUTHORITY["EPSG","9122"]], AUTHORITY["EPSG","4326"]] Origin = (6.500000000000000,60.000000000000000) Pixel Size = (0.012500000000000,-0.012500000000000) Image Structure Metadata: COMPRESSION=DEFLATE INTERLEAVE=BAND Corner Coordinates: Upper Left ( 6.5000000, 60.0000000) ( 6d30' 0.00"E, 60d 0' 0.00"N) Lower Left ( 6.5000000, 53.0000000) ( 6d30' 0.00"E, 53d 0' 0.00"N) Upper Right ( 14.5000000, 60.0000000) ( 14d30' 0.00"E, 60d 0' 0.00"N) Lower Right ( 14.5000000, 53.0000000) ( 14d30' 0.00"E, 53d 0' 0.00"N) Center ( 10.5000000, 56.5000000) ( 10d30' 0.00"E, 56d30' 0.00"N) Band 1 Block=128x128 Type=Float32, ColorInterp=Gray ERROR 1: ZLib return code (-3) ERROR 1: ZLib return code (-3) ... ERROR 1: ZLib return code (-3) ERROR 1: ZLib return code (-3) Checksum=53696 NoData Value=0 Overviews: 320x280, 160x140, 80x70, 40x35, 20x17 ERROR 1: ZLib return code (-3) ERROR 1: ZLib return code (-3) ... ERROR 1: ZLib return code (-3) ERROR 1: ZLib return code (-3) Overviews checksum: 62777, 64639, 65229, 71, 65201 bm4102@LinDevMap60:~$
Result on Big Endian (Solaris on Sparc):
[bm4102@utsira:/home/bm4102/gdal-1.11.2] gdalinfo geor:user/pw@machine,SDE_RDT_3778,24 -checksum Driver: GeoRaster/Oracle Spatial GeoRaster Files: none associated Size is 640, 560 Coordinate System is: GEOGCS["WGS 84", DATUM["WGS_1984", SPHEROID["WGS 84",6378137,298.257223563, AUTHORITY["EPSG","7030"]], AUTHORITY["EPSG","6326"]], PRIMEM["Greenwich",0, AUTHORITY["EPSG","8901"]], UNIT["degree",0.0174532925199433, AUTHORITY["EPSG","9122"]], AUTHORITY["EPSG","4326"]] Origin = (6.500000000000000,60.000000000000000) Pixel Size = (0.012500000000000,-0.012500000000000) Image Structure Metadata: COMPRESSION=DEFLATE INTERLEAVE=BAND Corner Coordinates: Upper Left ( 6.5000000, 60.0000000) Lower Left ( 6.5000000, 53.0000000) Upper Right ( 14.5000000, 60.0000000) Lower Right ( 14.5000000, 53.0000000) Center ( 10.5000000, 56.5000000) Band 1 Block=128x128 Type=Float32, ColorInterp=Gray Checksum=1553 NoData Value=0 Overviews: 320x280, 160x140, 80x70, 40x35, 20x17 Overviews checksum: 378, 87, 18, 4, 2 [bm4102@utsira:/home/bm4102/gdal-1.11.2]
Change History (8)
comment:1 by , 8 years ago
Owner: | changed from | to
---|
comment:2 by , 8 years ago
comment:3 by , 8 years ago
The questions above brought me on the right track:
The Oracle server runs on an MSB machine (AIX on Power7 hardware), the config log looked alright, zlib is taken from the system , none of CPL_MSB, CPL_LSB nor WORDS_BIGENDIAN was explicitly set.
Setting --with-zlib=internal and printing the bytes streamed into the MACRO PULLBYTE (frmts/zlib/inflate.c) to stderr clearly showed that the call to GDALSwapWords is the culprit: the stream is worked bytewise and the swapping disturbs the sequence of the data!
Removing the GDALSwapWords sequence from georaster_wrapper.cpp unconditionally keeps the stream inflatable and produces the correct result.
Regards Kai
comment:4 by , 8 years ago
Like Ivan explained, we store all binary numbers in raster blobs in Oracle in BIG ENDIAN always - irrespective of the nature of the platform it runs on (BIG or LITTLE ENDIAN).
This means that a GDAL build running on a LITTLE ENDIAN (=essentially Intel chips: Linux on intel, Windows, Solaris on Intel) must swap byte order. This is what the code does:
#ifndef CPL_MSB if( nCellSizeBits > 8 ) { int nWordSize = nCellSizeBits / 8; int nWordCount = nColumnBlockSize * nRowBlockSize * nBandBlockSize; GDALSwapWords( pabyBlockBuf, nWordSize, nWordCount, nWordSize ); } #endif
I.e unless CPL_MSB is defined, the georaster plugin will swap the bytes. That is the default behavior and is the way GDAL builds on common platforms (Linux, Windows) work.
To run GDAL on a BIG ENDIAN platform (= Solaris on Sparc, AIX on Power7, HP Tru64, Linux on Power ...) must not swap byte order. That is achieved by defining CPL_MSB for builds on those platforms. The presence of CPL_MSB will effectively skip the byte swapping in the code above.
comment:5 by , 8 years ago
Hi Kai,
Could you give us some details on how did you manage to build GDAL on AIX?
Which compiler and version did you used?
How does you ./configure command looks likes?
Did you have to make changes to port/cpl_port.h specific to _AIX?
If you want to write direct to me, my e-mail is on frtms/georaster source code .cpp/.h files header.
Thanks,
Ivan
comment:6 by , 8 years ago
Fixed on revision #33771
GDALSwapWords should be called after the buffer is uncompressed.
comment:7 by , 8 years ago
Resolution: | → fixed |
---|---|
Status: | new → closed |
Backported to 2.0
Please give it a try.
The Oracle server is running on the SPARC machine. Is that right?
Debugging that error would require quite a lot of effort to get access to a similar environment, so I would like to ask some questions first:
When you compiled GDAL and the georaster driver on the Intel machine...
As you may now, in GeoRaster the data is stored as Big Endian, no matter what platform the server is.
The GDAL that is running on the Intel Linux should call GDALSwapWords before the decompression unless there is something wrong with the way it was build.
http://trac.osgeo.org/gdal/browser/trunk/gdal/frmts/georaster/georaster_wrapper.cpp#L2011
Regards.