GDALDatasetCopyWholeRaster() hardcodes a very small size for intermediate buffer

[collinsb]

In rasterio.cpp, the call GDALDatasetCopyWholeRaster() sets the intermediate buffer to 10MB (line 1804). This buffer is then used (when interleaved) to scan down the rows of the image. For large images, the buffer then has very few pixels in the y-dimension and ends up thrashing to disk in cases where the block sizes of the two datasets do not line up. It seems that, like the "-wm" flag to set warp memory, there should be something for setting this buffer size in the case that more than 10 megabytes of memory is available.

[warmerdam]

The 10MB sizing is not just to keep memory use modest, it is also done this way so that progress counters (and UIs with cancel options) have a reasonable change to show progress and let the user interrupt the operation.

I do think we could stand to tune the rules. Perhaps allow working on chunks up to five times the "target buffer size" if the Y block size is such that one row of blocks doesn't fit in the 10MB.

I'm not against offering a config option for this but I much prefer to get the default rules operating fairly well.

[Jukka Rahkonen]

The source file is /trunk/gdal/gcore/rasterio.cpp

There is a comment

/* -------------------------------------------------------------------- */
3540	/*      What will our swath size be?                                    */
3541	/* -------------------------------------------------------------------- */

and code

{
3557	        CPLError( CE_Failure, CPLE_OutOfMemory,
3558	                "Failed to allocate %d*%d*%d byte swath buffer in\n"
3559	                "GDALDatasetCopyWholeRaster()",
3560	                nSwathCols, nSwathLines, nPixelSize );
3561	        return CE_Failure;
3562	    }

Does it mean that there is not a 10 MB hardcoded buffer size any more?

[Even Rouault]

There's a configuration option for it, GDAL_SWATH_SIZE, and now it's 1/4 of the GDAL_CACHEMAX (still defaulting to 10 MB is GDAL_CACHEMAX let to its default value)