gdaladdo slowness for large files

[warmerdam]

Mark L reports that running the commands:

gdaladdo -r average MWG_Tiled_600dpi_LZW.tif --config GDAL_CACHE_MAX 5000 2
gdaladdo -r average MWG_Tiled_600dpi_LZW.tif --config GDAL_CACHE_MAX 5000 4
gdaladdo -r average MWG_Tiled_600dpi_LZW.tif --config GDAL_CACHE_MAX 5000 8
gdaladdo -r average MWG_Tiled_600dpi_LZW.tif --config GDAL_CACHE_MAX 5000 16 32 64 128 256

On a one band paletted 8 bit file with 128x128 tiling in GeoTIFF that is 450000 x 430000 pixels took 4 weeks. I believe this is due to "tiff directory thrashing" as the driver switches back and forth between the primary image directory and the overview image directory.

I'm creating this bug as a TODO for myself to resolve this long standing scalability issue.

[osjonathan]

I'm getting this right now and it's really irksome. The file itself is only about 400MB and 95% of it is nodata, but it's really big in number of pixels.

My command:

  gdaladdo -r average process.tif 2 4 8 16 32 64 128 256

My gdalinfo for that file:

Driver: GTiff/GeoTIFF
Files: 1250_OSPlan_a5_tiled_but_not_overviewd.tif
Size is 450216, 441558
Coordinate System is:
PROJCS["OSGB 1936 / British National Grid",
    GEOGCS["OSGB 1936",
        DATUM["OSGB_1936",
            SPHEROID["Airy 1830",6377563.396,299.3249646000044,
                AUTHORITY["EPSG","7001"]],
            TOWGS84[446.448,-125.157,542.06,0.15,0.247,0.842,-20.489],
            AUTHORITY["EPSG","6277"]],
        PRIMEM["Greenwich",0],
        UNIT["degree",0.0174532925199433],
        AUTHORITY["EPSG","4277"]],
    PROJECTION["Transverse_Mercator"],
    PARAMETER["latitude_of_origin",49],
    PARAMETER["central_meridian",-2],
    PARAMETER["scale_factor",0.9996012717],
    PARAMETER["false_easting",400000],
    PARAMETER["false_northing",-100000],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    AUTHORITY["EPSG","27700"]]
Origin = (403000.018999999970000,304999.995999999980000)
Pixel Size = (0.115500000000000,-0.115500000000000)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  COMPRESSION=DEFLATE
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  (  403000.019,  304999.996) (  1d57'20.37"W, 52d38'32.58"N)
Lower Left  (  403000.019,  254000.047) (  1d57'22.02"W, 52d11' 1.81"N)
Upper Right (  454999.967,  304999.996) (  1d11'13.64"W, 52d38'22.57"N)
Lower Right (  454999.967,  254000.047) (  1d11'43.83"W, 52d10'51.97"N)
Center      (  428999.993,  279500.021) (  1d34'24.96"W, 52d24'44.47"N)
Band 1 Block=512x512 Type=Byte, ColorInterp=Gray

[Even Rouault]

You could potentially try the following workaround :

gdaladdo -ro -r average process.tif 2 (note the -ro flag, this will generate a process.tif.ovr file)
gdaladdo -ro -r average process.tif.ovr 2  (note the -ro flag, this will generate a process.tif.ovr.ovr file)

... repeated the number of times you need overviews.

Then you can combine all the files together :

gdal_translate process.tif process_with_ovr.tif -co COPY_SRC_OVERVIEWS YES -co COMPRESS=DEFLATE

The idea is that as the main file and its generated overviews are in different files, we will have two distinct tiff handles, and this will save the "tiff directory thrashing" effect.

[Jukka Rahkonen]

Have you tried the workaround, osjonathan? It feels like a working trick and the way to go if someone, sometimes implements a "gdaladdo_xxl" utility. It makes me think also if overviews in JPEG2000 format would be good for really big images. Unfortunately the best free library (OpenJPEG) is still quite slow.

[Even Rouault]

This ticket has been automatically closed because Trac is no longer used for GDAL bug tracking, since the project has migrated to GitHub. If you believe this ticket is still valid, you may file it to ​https://github.com/OSGeo/gdal/issues if it is not already reported there.

[Even Rouault]

Was recreated in github as ​https://github.com/OSGeo/gdal/issues/1442