Order of compressed overview levels drastically changes output size when doing levels separately

[kthujvu]

I noticed this when struggling to build overviews for a 90G, 1296001x417601 px, DEFLATEd GeoTIFF. I know it might be silly to process such a huge raster but hey, it's fun to try! :) While building individual levels always worked fine and in pretty reasonable time (~1 hour for a level), I could not get the whole range I wanted to complete (2 to 1024). Trying to incrementally do level by level into the same .ovr file resulted in some weirdness, which I show with a smaller sample below.

I did not actually look at the files, I only noticed the drastically different file sizes and decided that something is wrong.

Here is a smaller sample that shows the same oddities on my system: ​https://transfer.sh/6IZid/50n000e-20101117-gmted-std075.tif.xz (~30MB, gone in 2 weeks!). If it is gone, just message me to re-upload.

I reduced my commandline to the minimum to trigger this. It also happens similarly with LZW, I did not try other algorithms.

$ rm 50N000E_20101117_gmted_std075.tif.ovr
$ gdaladdo --config COMPRESS_OVERVIEW DEFLATE -ro 50N000E_20101117_gmted_std075.tif 2
0...10...20...30...40...50...60...70...80...90...100 - done.
$ gdaladdo --config COMPRESS_OVERVIEW DEFLATE -ro 50N000E_20101117_gmted_std075.tif 64
0...10...20...30...40...50...60...70...80...90...100 - done.
$ ls -l 50N000E_20101117_gmted_std075.tif.ovr
-rw-r--r-- 1 me users 9971477 Aug 10 11:26 50N000E_20101117_gmted_std075.tif.ovr

$ rm 50N000E_20101117_gmted_std075.tif.ovr
$ gdaladdo --config COMPRESS_OVERVIEW DEFLATE -ro 50N000E_20101117_gmted_std075.tif 64
0...10...20...30...40...50...60...70...80...90...100 - done.
$ gdaladdo --config COMPRESS_OVERVIEW DEFLATE -ro 50N000E_20101117_gmted_std075.tif 2
0...10...20...30...40...50...60...70...80...90...100 - done.
$ ls -l 50N000E_20101117_gmted_std075.tif.ovr
-rw-r--r-- 1 me users 49973 Aug 10 11:26 50N000E_20101117_gmted_std075.tif.ovr

Building level 2 first, then level 64 gives ~10 megabytes. Building level 64 first, then level 2 gives ~50 kilobytes.

Maybe irrelevant but also, doing 2->64 makes gdaladdo takes a while (maybe a second) to go from 90... to 100 in each step. It does run through instantly when doing 64->2.

This does not happen if I add the overviews in switched order in the same gdaladdo process:

$ rm 50N000E_20101117_gmted_std075.tif.ovr
$ gdaladdo --config COMPRESS_OVERVIEW DEFLATE -ro 50N000E_20101117_gmted_std075.tif 2 64
0...10...20...30...40...50...60...70...80...90...100 - done.
$ ls -l 50N000E_20101117_gmted_std075.tif.ovr
-rw-r--r-- 1 me users 9758804 Aug 10 11:41 50N000E_20101117_gmted_std075.tif.ovr
$ rm 50N000E_20101117_gmted_std075.tif.ovr
$ gdaladdo --config COMPRESS_OVERVIEW DEFLATE -ro 50N000E_20101117_gmted_std075.tif 64 2
0...10...20...30...40...50...60...70...80...90...100 - done.
$ ls -l 50N000E_20101117_gmted_std075.tif.ovr
-rw-r--r-- 1 me users 9758803 Aug 10 11:41 50N000E_20101117_gmted_std075.tif.ovr

This does not happen with a tiny sample:

$ wget -nv https://svn.osgeo.org/gdal/trunk/autotest/gdrivers/data/small_world.tif
2016-08-10 11:27:54 URL:https://svn.osgeo.org/gdal/trunk/autotest/gdrivers/data/small_world.tif [240574/240574] -> "small_world.tif" [1]

$ rm small_world.tif.ovr
$ gdaladdo --config COMPRESS_OVERVIEW DEFLATE -ro small_world.tif 2
0...10...20...30...40...50...60...70...80...90...100 - done.
$ gdaladdo --config COMPRESS_OVERVIEW DEFLATE -ro small_world.tif 64
0...10...20...30...40...50...60...70...80...90...100 - done.
$ ls -l small_world.tif.ovr
-rw-r--r-- 1 me users 28699 Aug 10 11:27 small_world.tif.ovr

$ rm small_world.tif.ovr
$ gdaladdo --config COMPRESS_OVERVIEW DEFLATE -ro small_world.tif 64
0...10...20...30...40...50...60...70...80...90...100 - done.
$ gdaladdo --config COMPRESS_OVERVIEW DEFLATE -ro small_world.tif 2
0...10...20...30...40...50...60...70...80...90...100 - done.
$ ls -l small_world.tif.ovr
-rw-r--r-- 1 me users 28699 Aug 10 11:27 small_world.tif.ovr

[Even Rouault]

In 35038:

External overview: fix incorrect overview building when generating from greater overview factors to lower ones, in compressed and single-band case (fixes #6617)

[Even Rouault]

In 35039:

External overview: fix incorrect overview building when generating from greater overview factors to lower ones, in compressed and single-band case (fixes #6617)

[Even Rouault]

This is a old problem that also affects previous GDAL releases, but I didn't bother backporting.

[Even Rouault]

In 35041:

Add missing include that lost got in the merge (fixes #6617)