#5067 closed defect (wontfix)
gdaladdo slowness for large files
Reported by: | warmerdam | Owned by: | warmerdam |
---|---|---|---|
Priority: | normal | Milestone: | closed_because_of_github_migration |
Component: | GDAL_Raster | Version: | unspecified |
Severity: | normal | Keywords: | GTiff |
Cc: |
Description
Mark L reports that running the commands:
gdaladdo -r average MWG_Tiled_600dpi_LZW.tif --config GDAL_CACHE_MAX 5000 2 gdaladdo -r average MWG_Tiled_600dpi_LZW.tif --config GDAL_CACHE_MAX 5000 4 gdaladdo -r average MWG_Tiled_600dpi_LZW.tif --config GDAL_CACHE_MAX 5000 8 gdaladdo -r average MWG_Tiled_600dpi_LZW.tif --config GDAL_CACHE_MAX 5000 16 32 64 128 256
On a one band paletted 8 bit file with 128x128 tiling in GeoTIFF that is 450000 x 430000 pixels took 4 weeks. I believe this is due to "tiff directory thrashing" as the driver switches back and forth between the primary image directory and the overview image directory.
I'm creating this bug as a TODO for myself to resolve this long standing scalability issue.
Change History (5)
comment:1 by , 11 years ago
comment:2 by , 11 years ago
You could potentially try the following workaround :
gdaladdo -ro -r average process.tif 2 (note the -ro flag, this will generate a process.tif.ovr file) gdaladdo -ro -r average process.tif.ovr 2 (note the -ro flag, this will generate a process.tif.ovr.ovr file)
... repeated the number of times you need overviews.
Then you can combine all the files together :
gdal_translate process.tif process_with_ovr.tif -co COPY_SRC_OVERVIEWS YES -co COMPRESS=DEFLATE
The idea is that as the main file and its generated overviews are in different files, we will have two distinct tiff handles, and this will save the "tiff directory thrashing" effect.
comment:3 by , 10 years ago
Have you tried the workaround, osjonathan? It feels like a working trick and the way to go if someone, sometimes implements a "gdaladdo_xxl" utility. It makes me think also if overviews in JPEG2000 format would be good for really big images. Unfortunately the best free library (OpenJPEG) is still quite slow.
comment:4 by , 5 years ago
Milestone: | → closed_because_of_github_migration |
---|---|
Resolution: | → wontfix |
Status: | new → closed |
This ticket has been automatically closed because Trac is no longer used for GDAL bug tracking, since the project has migrated to GitHub. If you believe this ticket is still valid, you may file it to https://github.com/OSGeo/gdal/issues if it is not already reported there.
I'm getting this right now and it's really irksome. The file itself is only about 400MB and 95% of it is nodata, but it's really big in number of pixels.
My command:
My gdalinfo for that file: