GDALaddo: LZW "Predictor = 2" option for OVR files, as exists for GeoTIFF files

[upperoso]

The option for LZW compression using Horizontal Differencing exists in gdal_translate. Horizontal differencing is particularly useful for 16-bit data when the high-order and low-order bytes are changing at different frequencies (think broad flat plateau as might exist in an elevation dataset).

Referring to the ASTER Global DEM Version 1 (ASTER GDEM V001), we find 22,603 tiles (_DEM.TIF extension) as well as 22,603 metadata tiles (_NUM.TIF extension; values refer to source data for building the DEM (# of satellites, hole-filling from SRTM, etc.)). As downloaded from the US or Japan, the user is confronted with 1.2 TB dataset. LZW compression will reduce this substantially, LZW with Horizontal Differencing even more so.

While LZW compression does reduce the size of the DEM from approximately 546 GB to approximately 265 GB, a large number of DEM tiles actually increase in size. Better and more consistent results require the "Predictor 2" option. Example (ASTGTM_N30E094):

RAW 25,963,980 bytes LZW 26,920,558 bytes LZWHD 12,928,732 bytes

The inclusion of the horizontal differencing to option to OVR files will increase their usefulness. The LZW Predictor=2 option should improve compression on any files greater than 8 bits / resel (pixel, cell, etc.).

At the end of this note, please refer to the list showing the original ASTER Global DEM as downloaded and compressed. The savings in disk space are substantial. The additional processing time to uncompress a LZW-HD GeoTIFF is all but unnoticeable (seat-of-the-pants metric).

Thank you for your consideration. I am happy to elaborate more on the method (in the TIFF spec. for a long, long time) and in its application (OVR files are quite useful, and will become more useful).

Best wishes,

David

ASTER GDEM V001

TIFF – Uncompressed (Original data) _DEM.TIF 586,863,840,204 bytes _NUM.TIF 586,863,840,204 bytes TOTAL 1,173,727,680,408 bytes

TIFF – Compressed (GDAL; LZW compression) _DEM.TIF 277,915,138 k _NUM.TIF Not Available ( I deleted the files! )

TIFF – Compressed (GDAL; Predictor=2) _DEM.TIF 121,420,311,014 bytes _NUM.TIF 36,625,546,640 bytes TOTAL 158,045,857,654 bytes

[Even Rouault]

Implemented in trunk (r18813). Test added in r18814

[upperoso]

Replying to rouault:

Implemented in trunk (r18813). Test added in r18814

The TIFF Tag ​Predictor is not being updated for Horizontal Differencing in gdaladdo with the enhancement as implemented.

The TIFF files from gdal_translate were compared to the TIFF files (OVR) from gdaladdo using exiftool ver 8.0.9.0 and AsTiffTagViewer ver 2.00. gdal_translate updates the TIFF Tag showing, as appropriate, Predictor = 2, or Horizontal differencing. gdal_addo does not do so.

The OVR files appear to be usabale as is. Perhaps the LZW compression embeds Predictor information in its output. The TIFF tag may be redundant, however it is called for in the TIFF specification.

Thanks!

David

[Even Rouault]

I'm not sure how you tested it.

Several possibilities :

• you did not grab the version with the patch
• you did not use the right option name. It is PREDICTOR_OVERVIEW
• you haven't deleted the overview if it existed before. PREDICTOR_OVERVIEW (as most other options) will be ignored for existing overviews.

Here's what it does for me :

$ gdaladdo -ro byte.tif 2 --config COMPRESS_OVERVIEW LZW --config PREDICTOR_OVERVIEW 2

$ tiffdump byte.tif.ovr
byte.tif.ovr:
Magic: 0x4949 <little-endian> Version: 0x2a
Directory 0: offset 8 (0x8) next 0 (0)
SubFileType (254) LONG (4) 1<1>
ImageWidth (256) SHORT (3) 1<10>
ImageLength (257) SHORT (3) 1<10>
BitsPerSample (258) SHORT (3) 1<8>
Compression (259) SHORT (3) 1<5>
Photometric (262) SHORT (3) 1<1>
SamplesPerPixel (277) SHORT (3) 1<1>
PlanarConfig (284) SHORT (3) 1<1>
Predictor (317) SHORT (3) 1<2>
TileWidth (322) SHORT (3) 1<128>
TileLength (323) SHORT (3) 1<128>
TileOffsets (324) LONG (4) 1<182>
TileByteCounts (325) LONG (4) 1<329>
SampleFormat (339) SHORT (3) 1<1>

Predictor (317) SHORT (3) 1<2> shows that it has worked.

Also confirmed by exiftool :

$  ./exiftool ../gdal/byte.tif.ovr 
ExifTool Version Number         : 8.10
File Name                       : byte.tif.ovr
Directory                       : ../gdal
File Size                       : 511 bytes
File Modification Date/Time     : 2010:02:16 02:04:50+01:00
File Permissions                : rw-r--r--
File Type                       : TIFF
MIME Type                       : image/tiff
Exif Byte Order                 : Little-endian (Intel, II)
Subfile Type                    : Reduced-resolution image
Image Width                     : 10
Image Height                    : 10
Bits Per Sample                 : 8
Compression                     : LZW
Photometric Interpretation      : BlackIsZero
Samples Per Pixel               : 1
Planar Configuration            : Chunky
Predictor                       : Horizontal differencing
Tile Width                      : 128
Tile Length                     : 128
Tile Offsets                    : 182
Tile Byte Counts                : 329
Sample Format                   : Unsigned
Image Size                      : 10x10

[upperoso]

Right. Thanks. Using the 1.7.1 snapshot, I implemented the change for LZW w/ HD. Whatever inconsistencies were apparent earlier today are gone. The only change was a reboot of a Windows XP SP3 machine. Thanks for double-checking with me. Everything is working *exactly* as I expected it should and expect it should.

Sorry for the trouble.

Best, David