id,summary,reporter,owner,description,type,status,priority,milestone,component,version,severity,resolution,keywords,cc 6617,Order of compressed overview levels drastically changes output size when doing levels separately,kthujvu,warmerdam,"I noticed this when struggling to build overviews for a 90G, 1296001x417601 px, DEFLATEd GeoTIFF. I know it might be silly to process such a huge raster but hey, it's fun to try! :) While building individual levels always worked fine and in pretty reasonable time (~1 hour for a level), I could not get the whole range I wanted to complete (2 to 1024). Trying to incrementally do level by level into the same .ovr file resulted in some weirdness, which I show with a smaller sample below. I did not actually look at the files, I only noticed the drastically different file sizes and decided that something is wrong. Here is a smaller sample that shows the same oddities on my system: https://transfer.sh/6IZid/50n000e-20101117-gmted-std075.tif.xz (~30MB, gone in 2 weeks!). If it is gone, just message me to re-upload. I reduced my commandline to the minimum to trigger this. It also happens similarly with LZW, I did not try other algorithms. {{{ $ rm 50N000E_20101117_gmted_std075.tif.ovr $ gdaladdo --config COMPRESS_OVERVIEW DEFLATE -ro 50N000E_20101117_gmted_std075.tif 2 0...10...20...30...40...50...60...70...80...90...100 - done. $ gdaladdo --config COMPRESS_OVERVIEW DEFLATE -ro 50N000E_20101117_gmted_std075.tif 64 0...10...20...30...40...50...60...70...80...90...100 - done. $ ls -l 50N000E_20101117_gmted_std075.tif.ovr -rw-r--r-- 1 me users 9971477 Aug 10 11:26 50N000E_20101117_gmted_std075.tif.ovr $ rm 50N000E_20101117_gmted_std075.tif.ovr $ gdaladdo --config COMPRESS_OVERVIEW DEFLATE -ro 50N000E_20101117_gmted_std075.tif 64 0...10...20...30...40...50...60...70...80...90...100 - done. $ gdaladdo --config COMPRESS_OVERVIEW DEFLATE -ro 50N000E_20101117_gmted_std075.tif 2 0...10...20...30...40...50...60...70...80...90...100 - done. $ ls -l 50N000E_20101117_gmted_std075.tif.ovr -rw-r--r-- 1 me users 49973 Aug 10 11:26 50N000E_20101117_gmted_std075.tif.ovr }}} Building level 2 first, then level 64 gives ~10 megabytes. Building level 64 first, then level 2 gives ~50 kilobytes. Maybe irrelevant but also, doing 2->64 makes gdaladdo takes a while (maybe a second) to go from 90... to 100 in each step. It does run through instantly when doing 64->2. This does not happen if I add the overviews in switched order in the same gdaladdo process: {{{ $ rm 50N000E_20101117_gmted_std075.tif.ovr $ gdaladdo --config COMPRESS_OVERVIEW DEFLATE -ro 50N000E_20101117_gmted_std075.tif 2 64 0...10...20...30...40...50...60...70...80...90...100 - done. $ ls -l 50N000E_20101117_gmted_std075.tif.ovr -rw-r--r-- 1 me users 9758804 Aug 10 11:41 50N000E_20101117_gmted_std075.tif.ovr $ rm 50N000E_20101117_gmted_std075.tif.ovr $ gdaladdo --config COMPRESS_OVERVIEW DEFLATE -ro 50N000E_20101117_gmted_std075.tif 64 2 0...10...20...30...40...50...60...70...80...90...100 - done. $ ls -l 50N000E_20101117_gmted_std075.tif.ovr -rw-r--r-- 1 me users 9758803 Aug 10 11:41 50N000E_20101117_gmted_std075.tif.ovr }}} This does not happen with a tiny sample: {{{ $ wget -nv https://svn.osgeo.org/gdal/trunk/autotest/gdrivers/data/small_world.tif 2016-08-10 11:27:54 URL:https://svn.osgeo.org/gdal/trunk/autotest/gdrivers/data/small_world.tif [240574/240574] -> ""small_world.tif"" [1] $ rm small_world.tif.ovr $ gdaladdo --config COMPRESS_OVERVIEW DEFLATE -ro small_world.tif 2 0...10...20...30...40...50...60...70...80...90...100 - done. $ gdaladdo --config COMPRESS_OVERVIEW DEFLATE -ro small_world.tif 64 0...10...20...30...40...50...60...70...80...90...100 - done. $ ls -l small_world.tif.ovr -rw-r--r-- 1 me users 28699 Aug 10 11:27 small_world.tif.ovr $ rm small_world.tif.ovr $ gdaladdo --config COMPRESS_OVERVIEW DEFLATE -ro small_world.tif 64 0...10...20...30...40...50...60...70...80...90...100 - done. $ gdaladdo --config COMPRESS_OVERVIEW DEFLATE -ro small_world.tif 2 0...10...20...30...40...50...60...70...80...90...100 - done. $ ls -l small_world.tif.ovr -rw-r--r-- 1 me users 28699 Aug 10 11:27 small_world.tif.ovr }}} ",defect,closed,normal,2.1.2,GDAL_Raster,2.1.0,normal,fixed,"overviews, compression, gdaladdo",