GDALRasterBand::IRasterIO fails to read data from dataset when block cache is too small

[Even Rouault]

To demonstrate the problem, run :

GDAL_CACHEMAX=0 gdal_translate ../autotest/gcore/data/utmsmall.tif temp.tif -co TILED=YES -co BLOCKXSIZE=16 -co BLOCKYSIZE=16

The resulting checksum will be wrong (46979) instead of 50054.

In rasterio.cpp, we are in the case of line 171. The cache size is such that blocks get flushed out several times while iterating over iBufYOff, but when they are reloaded their content is not read back from the dataset. The GDAL_CACHEMAX=0 value is of course very particular, but I think we could also hit that case when writing buffers very large in the X direction, in fact if blockByteSize * (nXSize / nBlockXSize) > GDAL_CACHEMAX.

So, if nBlockXSize = nBlockYSize = 256 and GDAL_CACHEMAX = 40 MB, we have the critical nXSize ~= 155 000

The following code also demonstrates the problem:

import gdal
one_ds =  gdal.GetDriverByName("GTiff").Create("ones.tif", 200000, 256,  1, options = [ 'TILED=YES', 'BLOCKXSIZE=256', 'BLOCKYSIZE=256'])
one_ds.GetRasterBand(1).Fill(1)
ds =  gdal.GetDriverByName("GTiff").Create("temp.tif", 200000, 256,  1, options = [ 'TILED=YES', 'BLOCKXSIZE=256', 'BLOCKYSIZE=256'])
data = one_ds.ReadRaster(0, 0, 200000, 256)
ds.WriteRaster(0, 0, 200000, 256, data)
one_ds = None
ds = None

In DEBUG mode, we have the following complaint : "GDAL: Potential thrashing on band 1 of ones.tif"

The solution is to rearrange a bit the way we write the block such that we fill it up completely instead of loading it each time. That is to say that we have a sub-loop on y at the deeper level. This will save quite a few call to GetLockedBlockRef at the same time.

[Even Rouault]

Fixed in trunk in r14844. Test added in r14845

[Even Rouault]

Applied in branches/1.5 in r14863