Opened 16 years ago
Closed 16 years ago
#2457 closed defect (fixed)
GDALRasterBand::IRasterIO fails to read data from dataset when block cache is too small
Reported by: | Even Rouault | Owned by: | warmerdam |
---|---|---|---|
Priority: | normal | Milestone: | 1.5.3 |
Component: | GDAL_Raster | Version: | 1.5.0 |
Severity: | normal | Keywords: | |
Cc: |
Description
To demonstrate the problem, run :
GDAL_CACHEMAX=0 gdal_translate ../autotest/gcore/data/utmsmall.tif temp.tif -co TILED=YES -co BLOCKXSIZE=16 -co BLOCKYSIZE=16
The resulting checksum will be wrong (46979) instead of 50054.
In rasterio.cpp, we are in the case of line 171. The cache size is such that blocks get flushed out several times while iterating over iBufYOff, but when they are reloaded their content is not read back from the dataset. The GDAL_CACHEMAX=0 value is of course very particular, but I think we could also hit that case when writing buffers very large in the X direction, in fact if blockByteSize * (nXSize / nBlockXSize) > GDAL_CACHEMAX.
So, if nBlockXSize = nBlockYSize = 256 and GDAL_CACHEMAX = 40 MB, we have the critical nXSize ~= 155 000
The following code also demonstrates the problem:
import gdal one_ds = gdal.GetDriverByName("GTiff").Create("ones.tif", 200000, 256, 1, options = [ 'TILED=YES', 'BLOCKXSIZE=256', 'BLOCKYSIZE=256']) one_ds.GetRasterBand(1).Fill(1) ds = gdal.GetDriverByName("GTiff").Create("temp.tif", 200000, 256, 1, options = [ 'TILED=YES', 'BLOCKXSIZE=256', 'BLOCKYSIZE=256']) data = one_ds.ReadRaster(0, 0, 200000, 256) ds.WriteRaster(0, 0, 200000, 256, data) one_ds = None ds = None
In DEBUG mode, we have the following complaint : "GDAL: Potential thrashing on band 1 of ones.tif"
The solution is to rearrange a bit the way we write the block such that we fill it up completely instead of loading it each time. That is to say that we have a sub-loop on y at the deeper level. This will save quite a few call to GetLockedBlockRef at the same time.
Change History (2)
comment:1 by , 16 years ago
comment:2 by , 16 years ago
Milestone: | → 1.5.3 |
---|---|
Resolution: | → fixed |
Status: | new → closed |
Applied in branches/1.5 in r14863
Fixed in trunk in r14844. Test added in r14845