| Version 22 (modified by , 3 weeks ago) ( diff ) |
|---|
The gdalwarp utility
gdalwarp is an image mosaicing, reprojection and warping utility. The program can reproject to any supported projection, and can also apply GCPs stored with the image if the image is "raw" with control information.
The official documentation for the
gdalwarputility is http://www.gdal.org/gdalwarp.html.
Error allocating memory
In some cases using --config GDAL_CACHEMAX xxx -wm xxx can be detrimental. A symptom of this is:
ERROR 2: Out of memory allocating 365425784 byte destination buffer.
Our understanding is that 32 bit processes are frequently subject to memory fragmentation problems and so even in a process with - in theory - 2GB of heap RAM space available it is still often difficult to allocate large blocks of memory. In this case with -wm 500 set the main warp buffers can be quite large (364MB for the example) and it seems it has failed to allocate the buffer. Use more modest buffers (or not use the options at all) or else get a 64 bit executable for gdalwarp.
Source:
Warp and Cache Memory: Technical Details
The GDAL_CACHEMAX affects the amount of space reserved for the low
level io buffers. When blocks are read from disk, or written to disk,
they are cached in a global block cache by the GDALRasterBlock class.
Once this cache exceeds GDAL_CACHEMAX old blocks are flushed from the
cache.
You can think of this as primarily an IO cache, and it mostly benefits you when you might need to read or write file blocks more than once. This could occur, for instance, in a scanline oriented input file which is processed in multiple chunks (horizontally) by gdalwarp.
The -wm flag affects the warping algorithm. The warper will total up the memory required to hold the input and output image arrays and any auxilary masking arrays and if they are larger than the "warp memory" allowed it will subdivide the chunk into smaller chunks and try again.
If the -wm value is very small there is some extra overhead in doing many small chunks so setting it larger is better but it is a matter of diminishing returns.
Overall, there is only so much more memory can do for you.
To get a better sense of how things are working, you might want to try running in debug mode. For instance:
gdalwarp --debug on abc.tif def.tif
When things shut down you may see messages like:
GDAL: 224 block reads on 32 block band 1 of utm.tif.
In this case it is saying that band 1 of utm.tif has 32 blocks, but that 224 block reads were done, implying that lots of data was having to be re-read, presumably because of a limited io cache.
You will also see messages like:
GDAL: GDALWarpKernel()::GWKNearestNoMasksByte() Src=0,0,512x512 Dst=0,0,512x512
The Src/Dst windows show you the "chunk size" being used. In this case my whole image which is very small. If you find things are being broken into a lot of chunks increasing -wm may help somewhat.
But far more important than memory are ensuring you are going through
an optimized path in the warper. If you ever see it reporting
GDALWarpKernel()::GWKGeneralCase() you know things will be relatively
slow. Basically, the fastest situations are nearest neighbour resampling
on 8bit data with no nodata or alpha masking in effect.
Sources
[GeoTIFF output] Use -co TILED=YES when possible
Due to the way gdalwarp proceeds, when generating a huge output file (width > 100,000 pixels for example), you should consider producing a tiled GeoTIFF file if it is an option for you (some software will only read strip TIFF files). If you have a lot of RAM, increasing the maximum cache size and the working buffers of the warping algorithm as explained above might also help if you generate a strip TIFF file.
