Changes between Initial Version and Version 1 of UserDocs/GdalWarp

Timestamp:

Aug 1, 2007, 11:01:47 AM (17 years ago)

Author:

maphew

Comment:

how -wm and CACHEMAX work

UserDocs/GdalWarp

@@ +1 @@
+
+
+=== Will increasing RAM increase the speed of gdalwarp? ===
+
+Adding ram will almost certainly increase the speed.  That’s not at all the same as saying that it is worth it, or that the speed increase will be significant. Disks are the slowest part of the process.
+
+By default gdalwarp won't take much advantage of RAM. Using the flag "-wm 500" will operate on 500MB chunks at a time which is better than the default. To increase the io block cache size may also help.  This can be done on the command like:
+{{{
+gdalwarp --config GDAL_CACHEMAX 500 -wm 500 ...
+}}}
+
+This uses 500MB of RAM for read/write caching, and 500MB of RAM for working buffers during the warp.  Beyond that it is doubtful more memory will make a substantial difference. 
+
+Check CPU usage while gdalwarp is running.  If it is substantially less than 100% then you know things are IO bound.  Otherwise they are CPU bound. 
+
+
+==== Warp and Cache Memory: Technical Details ====
+
+The GDAL_CACHEMAX affects the amount of space reserved for the low
+level io buffers.  When blocks are read from disk, or written to disk,
+they are cached in a global block cache by the `GDALRasterBlock` class.
+Once this cache exceeds `GDAL_CACHEMAX` old blocks are flushed from the
+cache.
+
+You can think of this as primarily an IO cache, and it mostly benefits
+you when you might need to read or write file blocks more than once.
+This could occur, for instance, in a scanline oriented input file which
+is processed in multiple chunks (horizontally) by gdalwarp.
+
+The -wm flag affects the warping algorithm.   The warper will total up
+the memory required to hold the input and output image arrays and any
+auxilary masking arrays and if they are larger than the "warp memory"
+allowed it will subdivide the chunk into smaller chunks and try again.
+
+If the -wm value is very small there is some extra overhead in doing many
+small chunks so setting it larger is better but it is a matter of
+diminishing returns.
+
+Overall, there is only so much more memory can do for you.
+
+To get a better sense of how things are working, you might want to
+try running in debug mode.  For instance:
+{{{
+  gdalwarp --debug on abc.tif def.tif
+}}}
+When things shut down you may see messages like:
+{{{
+  GDAL: 224 block reads on 32 block band 1 of utm.tif.
+}}}
+In this case it is saying that band 1 of utm.tif has 32 blocks, but
+that 224 block reads were done, implying that lots of data was
+having to be re-read, presumably because of a limited io cache.
+
+You will also see messages like:
+{{{
+GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
+Src=0,0,512x512 Dst=0,0,512x512
+}}}
+The Src/Dst windows show you the "chunk size" being used.  In this
+case my whole image which is very small.  If you find things are being
+broken into a lot of chunks increasing -wm may help somewhat.
+
+But far more important than memory are ensuring you are going through
+an optimized path in the warper.  If you ever see it reporting
+`GDALWarpKernel()::GWKGeneralCase()` you know things will be relatively
+slow.  Basically, the fastest situations are nearest neighbour resampling
+on 8bit data with no nodata or alpha masking in effect.
+
+-----
+Sources:
+   * [Gdal-dev] Will increasing the RAM on a server increase the speed of gdalwarp? - http://lists.maptools.org/pipermail/gdal-dev/2007-July/013566.html
+