OVR and RRD files cannot be larger than 4gb

[gaopeng]

I have a large tiff dataset on which I try to build pyramids. The pyramids go until size is 3.99 GB. Then the ArcGIS gp tool (application) crashes. I can load the data and the pyramids do not look correct.

The same problem exists for .rrd pyramid files. They also have the 4 GB size limit and are incorrect if need to be larger than that.

Here is the call stack when it crashed on tiff ovr.

>	gdal16d.dll!GDALRasterBand::FlushBlock(int nXBlockOff=3, int nYBlockOff=65)  Line 957 + 0x3 bytes	C++
 	gdal16d.dll!GDALRasterBlock::FlushCacheBlock()  Line 171	C++
 	gdal16d.dll!GDALFlushCacheBlock()  Line 134	C++
 	gdal16d.dll!GDALRasterBlock::Internalize()  Line 389	C++
 	gdal16d.dll!GDALRasterBand::GetLockedBlockRef(int nXBlockOff=671, int nYBlockOff=1, int bJustInitialize=0)  Line 1219 + 0x8 bytes	C++
 	gdal16d.dll!GTiffRasterBand::IReadBlock(int nBlockXOff=671, int nBlockYOff=1, void * pImage=0x2e44dd48)  Line 678 + 0x23 bytes	C++
 	gdal16d.dll!GDALRasterBand::GetLockedBlockRef(int nXBlockOff=671, int nYBlockOff=1, int bJustInitialize=0)  Line 1234 + 0x24 bytes	C++
 	gdal16d.dll!GDALRasterBand::IRasterIO(GDALRWFlag eRWFlag=GF_Read, int nXOff=0, int nYOff=128, int nXSize=200000, int nYSize=128, void * pData=0x21980040, int nBufXSize=200000, int nBufYSize=128, GDALDataType eBufType=GDT_Float32, int nPixelSpace=4, int nLineSpace=800000)  Line 217 + 0x17 bytes	C++
 	gdal16d.dll!GDALRasterBand::RasterIO(GDALRWFlag eRWFlag=GF_Read, int nXOff=0, int nYOff=128, int nXSize=200000, int nYSize=128, void * pData=0x21980040, int nBufXSize=200000, int nBufYSize=128, GDALDataType eBufType=GDT_Float32, int nPixelSpace=4, int nLineSpace=800000)  Line 243	C++
 	gdal16d.dll!GDALRegenerateOverviews(void * hSrcBand=0x05899050, int nOverviewCount=9, void * * pahOvrBands=0x28593ac0, const char * pszResampling=0x19d4a7e8, int (double, const char *, void *)* pfnProgress=0x0c79474b, void * pProgressData=0x2cde6078)  Line 1081	C++
 	gdal16d.dll!GTIFFBuildOverviews(const char * pszFilename=0x055f9e90, int nBands=3, GDALRasterBand * * papoBandList=0x055fbd00, int nOverviews=9, int * panOverviewList=0x0557f820, const char * pszResampling=0x19d4a7e8, int (double, const char *, void *)* pfnProgress=0x19caf800, void * pProgressData=0x0012e4f4)  Line 636 + 0x2a bytes	C++
 	gdal16d.dll!GDALDefaultOverviews::BuildOverviews(const char * pszBasename=0x00000000, const char * pszResampling=0x19d4a7e8, int nOverviews=9, int * panOverviewList=0x0012e680, int nBands=3, int * panBandList=0x0012e56c, int (double, const char *, void *)* pfnProgress=0x19caf800, void * pProgressData=0x0012e4f4)  Line 566 + 0x30 bytes	C++
 	gdal16d.dll!GDALDataset::IBuildOverviews(const char * pszResampling=0x19d4a7e8, int nOverviews=9, int * panOverviewList=0x0012e680, int nListBands=3, int * panBandList=0x0012e56c, int (double, const char *, void *)* pfnProgress=0x19caf800, void * pProgressData=0x0012e4f4)  Line 1373 + 0x29 bytes	C++
 	gdal16d.dll!GTiffDataset::IBuildOverviews(const char * pszResampling=0x19d4a7e8, int nOverviews=9, int * panOverviewList=0x0012e680, int nBands=3, int * panBandList=0x0012e56c, int (double, const char *, void *)* pfnProgress=0x19caf800, void * pProgressData=0x0012e4f4)  Line 3037 + 0x27 bytes	C++
 	gdal16d.dll!GDALDataset::BuildOverviews(const char * pszResampling=0x19d4a7e8, int nOverviews=9, int * panOverviewList=0x0012e680, int nListBands=3, int * panBandList=0x0012e56c, int (double, const char *, void *)* pfnProgress=0x19caf800, void * pProgressData=0x0012e4f4)  Line 1325 + 0x29 bytes	C++
 	GdalRasterDB.dll!GdlRasterDataset::BuildCompressedPyramid(long maxLevel=9, rstResamplingTypes resampling=RSP_NearestNeighbor, esriRasterCompressionType compressionType=-1, long quality=75)  Line 1745 + 0x3b bytes	C++
 	GpDataManagementFunctions.dll!GPBuildPyramids::Execute(IArray * pParameters=0x0589db94, ITrackCancel * pTrackCancel=0x0589e07c, IGPEnvironmentManager * pEnvMgr=0x05891a74, IGPMessages * pMessages=0x0589b38c)  Line 246 + 0x43 bytes	C++

[warmerdam]

A preliminary test with a 135000x135000 base file, made read-only and then running "gdaladdo big.tif 2" resulted in:

warmerda@gdal64[20]% ls -l big*
-r--r--r-- 1 warmerda warmerda  399788622 2010-02-01 23:34 big.tif
-rw-r--r-- 1 warmerda warmerda 4572057892 2010-02-01 23:51 big.tif.ovr

So creating a larger than 4GB file seems to have worked. This is with GDAL trunk, on linux.

I will try again tomorrow on windows. It takes quite a while!

[warmerdam]

I have confirmed the problem does occur with the 1.6-esri branch on win32, but not with trunk on win32. I presume there has been some fixing related to forcing bigtiff mode that did not make it into 1.6-esri. I will investigate on that basis.

[warmerdam]

I have ported some improvements to gt_overview.cpp from trunk into 1.6-esri (r18703). Now TIFF overviews should usually be done in BigTIFF is there is a significant chance of the .ovr file growing larger than 4GB.

So far, I'm not seeing problems with imagine format overviews on large files. It appears an .ige file is created. I will try a few more experiments.

[warmerdam]

I have tried a few more experiments, and been unable to reproduce a crash with large .rrd files. Can you reproduce the problem with gdaladdo?

[gaopeng]

OVR works for the original test case (uncompressed), but crashes for JPEG compression with quality 50.

RRD does not crash. But the resulting overiews are incorrect. For example, the generated overviews for a RGB image are grayscale.

[warmerdam]

Peng Gao writes:

Here is what I found on creating large RRD, when testing a 79G IMG image.

• The following condition (bCreateLargeRaster) is false so a separate .rrd is created, but it's too small, and overview generation failed.

      if( (psInfo->nEndOfFile 
           + (nOXSize * (double) nOYSize)
           * (HFAGetDataTypeBits(nDataType) / 8)) > 2000000000.0 )
          bCreateLargeRaster = TRUE;
  
      if( bCreateLargeRaster )
      {
          if( !HFACreateSpillStack( psInfo, nOXSize, nOYSize, 1, 
                                    64, nDataType, 
                                    &nValidFlagsOffset, &nDataOffset ) )
  	{
  	    return -1;
  	}
      }
  

• Once the flag bCreateLargeRaster is forced to true, a small .rrd is created, but no .rde. The overviews are actually written to the existing .ige, which is not desirable.

The behavior I am looking for is:

• If the uncompressed size of the entire overviews, including all bands, and all levels, is larger than 2G, create a large file.

• Always create overviews in a separate .rrd (.rde) file.

[warmerdam]

Gao,

I'm not really clear on why bCreateLargeRaster would have been determined to be false. A level 2x overview for a 79GB images should still be around 20GB.

I'm wondering if it is only some of the lower res overviews that are being treated as if they will fit in the .rrd file, but if the problem is that nEndOfFile does not reflect the size of previous overviews if they haven't been written yet. That is, if we create a 1.9GB overview, and a 500MB overview in quick succession before either are written it will not be obvious that cumulatively they are going to end up being more than 2GB. If this is the case the bug might only trigger in some fairly particular sizes. That might explain why I spent so much fruitless time trying to reproduce this bug.

[gaopeng]

The first level overview created is at 4x, and the size calculation only uses 1 band. So the resulting size is 79G/3/16, which is about 1.7G.

I am not clear how nEndOfFile works. Is it for the base image or the ovr?

[warmerdam]

OK, several problems now confirmed.

1) Creating jpeg compressed overviews on tiff with 1.6-esri does produce the expected error. Presumably the .ovr file is not created as big tiff when compression is used since it is hard to predict how large the file will get.

2) Creating .rrd files on a large file where each band overview is less than 2GB does not result in production of a spill file - at least for my test case. And the resulting >2GB file ends up having "2GB barrier wrap" issues so the imagery is corrupt - possibly the cause of apparent greyscale imagery?

Working on resolutions.