Opened 11 years ago

Closed 11 years ago

#2457 closed defect (fixed)

GDALRasterBand::IRasterIO fails to read data from dataset when block cache is too small

Reported by: Even Rouault Owned by: warmerdam
Priority: normal Milestone: 1.5.3
Component: GDAL_Raster Version: 1.5.0
Severity: normal Keywords:
Cc:

Description

To demonstrate the problem, run :

GDAL_CACHEMAX=0 gdal_translate ../autotest/gcore/data/utmsmall.tif temp.tif -co TILED=YES -co BLOCKXSIZE=16 -co BLOCKYSIZE=16

The resulting checksum will be wrong (46979) instead of 50054.

In rasterio.cpp, we are in the case of line 171. The cache size is such that blocks get flushed out several times while iterating over iBufYOff, but when they are reloaded their content is not read back from the dataset. The GDAL_CACHEMAX=0 value is of course very particular, but I think we could also hit that case when writing buffers very large in the X direction, in fact if blockByteSize * (nXSize / nBlockXSize) > GDAL_CACHEMAX.

So, if nBlockXSize = nBlockYSize = 256 and GDAL_CACHEMAX = 40 MB, we have the critical nXSize ~= 155 000

The following code also demonstrates the problem:

import gdal
one_ds =  gdal.GetDriverByName("GTiff").Create("ones.tif", 200000, 256,  1, options = [ 'TILED=YES', 'BLOCKXSIZE=256', 'BLOCKYSIZE=256'])
one_ds.GetRasterBand(1).Fill(1)
ds =  gdal.GetDriverByName("GTiff").Create("temp.tif", 200000, 256,  1, options = [ 'TILED=YES', 'BLOCKXSIZE=256', 'BLOCKYSIZE=256'])
data = one_ds.ReadRaster(0, 0, 200000, 256)
ds.WriteRaster(0, 0, 200000, 256, data)
one_ds = None
ds = None

In DEBUG mode, we have the following complaint : "GDAL: Potential thrashing on band 1 of ones.tif"

The solution is to rearrange a bit the way we write the block such that we fill it up completely instead of loading it each time. That is to say that we have a sub-loop on y at the deeper level. This will save quite a few call to GetLockedBlockRef? at the same time.

Change History (2)

comment:1 Changed 11 years ago by Even Rouault

Fixed in trunk in r14844. Test added in r14845

comment:2 Changed 11 years ago by Even Rouault

Milestone: 1.5.3
Resolution: fixed
Status: newclosed

Applied in branches/1.5 in r14863

Note: See TracTickets for help on using tickets.