Opened 10 years ago

Closed 10 years ago

#3371 closed defect (fixed)

OVR and RRD files cannot be larger than 4gb

Reported by: gaopeng Owned by: warmerdam
Priority: normal Milestone: 1.7.2
Component: GDAL_Raster Version: 1.6.1
Severity: normal Keywords: overview
Cc:

Description (last modified by warmerdam)

I have a large tiff dataset on which I try to build pyramids. The pyramids go until size is 3.99 GB. Then the ArcGIS gp tool (application) crashes. I can load the data and the pyramids do not look correct.

The same problem exists for .rrd pyramid files. They also have the 4 GB size limit and are incorrect if need to be larger than that.

Here is the call stack when it crashed on tiff ovr.

>	gdal16d.dll!GDALRasterBand::FlushBlock(int nXBlockOff=3, int nYBlockOff=65)  Line 957 + 0x3 bytes	C++
 	gdal16d.dll!GDALRasterBlock::FlushCacheBlock()  Line 171	C++
 	gdal16d.dll!GDALFlushCacheBlock()  Line 134	C++
 	gdal16d.dll!GDALRasterBlock::Internalize()  Line 389	C++
 	gdal16d.dll!GDALRasterBand::GetLockedBlockRef(int nXBlockOff=671, int nYBlockOff=1, int bJustInitialize=0)  Line 1219 + 0x8 bytes	C++
 	gdal16d.dll!GTiffRasterBand::IReadBlock(int nBlockXOff=671, int nBlockYOff=1, void * pImage=0x2e44dd48)  Line 678 + 0x23 bytes	C++
 	gdal16d.dll!GDALRasterBand::GetLockedBlockRef(int nXBlockOff=671, int nYBlockOff=1, int bJustInitialize=0)  Line 1234 + 0x24 bytes	C++
 	gdal16d.dll!GDALRasterBand::IRasterIO(GDALRWFlag eRWFlag=GF_Read, int nXOff=0, int nYOff=128, int nXSize=200000, int nYSize=128, void * pData=0x21980040, int nBufXSize=200000, int nBufYSize=128, GDALDataType eBufType=GDT_Float32, int nPixelSpace=4, int nLineSpace=800000)  Line 217 + 0x17 bytes	C++
 	gdal16d.dll!GDALRasterBand::RasterIO(GDALRWFlag eRWFlag=GF_Read, int nXOff=0, int nYOff=128, int nXSize=200000, int nYSize=128, void * pData=0x21980040, int nBufXSize=200000, int nBufYSize=128, GDALDataType eBufType=GDT_Float32, int nPixelSpace=4, int nLineSpace=800000)  Line 243	C++
 	gdal16d.dll!GDALRegenerateOverviews(void * hSrcBand=0x05899050, int nOverviewCount=9, void * * pahOvrBands=0x28593ac0, const char * pszResampling=0x19d4a7e8, int (double, const char *, void *)* pfnProgress=0x0c79474b, void * pProgressData=0x2cde6078)  Line 1081	C++
 	gdal16d.dll!GTIFFBuildOverviews(const char * pszFilename=0x055f9e90, int nBands=3, GDALRasterBand * * papoBandList=0x055fbd00, int nOverviews=9, int * panOverviewList=0x0557f820, const char * pszResampling=0x19d4a7e8, int (double, const char *, void *)* pfnProgress=0x19caf800, void * pProgressData=0x0012e4f4)  Line 636 + 0x2a bytes	C++
 	gdal16d.dll!GDALDefaultOverviews::BuildOverviews(const char * pszBasename=0x00000000, const char * pszResampling=0x19d4a7e8, int nOverviews=9, int * panOverviewList=0x0012e680, int nBands=3, int * panBandList=0x0012e56c, int (double, const char *, void *)* pfnProgress=0x19caf800, void * pProgressData=0x0012e4f4)  Line 566 + 0x30 bytes	C++
 	gdal16d.dll!GDALDataset::IBuildOverviews(const char * pszResampling=0x19d4a7e8, int nOverviews=9, int * panOverviewList=0x0012e680, int nListBands=3, int * panBandList=0x0012e56c, int (double, const char *, void *)* pfnProgress=0x19caf800, void * pProgressData=0x0012e4f4)  Line 1373 + 0x29 bytes	C++
 	gdal16d.dll!GTiffDataset::IBuildOverviews(const char * pszResampling=0x19d4a7e8, int nOverviews=9, int * panOverviewList=0x0012e680, int nBands=3, int * panBandList=0x0012e56c, int (double, const char *, void *)* pfnProgress=0x19caf800, void * pProgressData=0x0012e4f4)  Line 3037 + 0x27 bytes	C++
 	gdal16d.dll!GDALDataset::BuildOverviews(const char * pszResampling=0x19d4a7e8, int nOverviews=9, int * panOverviewList=0x0012e680, int nListBands=3, int * panBandList=0x0012e56c, int (double, const char *, void *)* pfnProgress=0x19caf800, void * pProgressData=0x0012e4f4)  Line 1325 + 0x29 bytes	C++
 	GdalRasterDB.dll!GdlRasterDataset::BuildCompressedPyramid(long maxLevel=9, rstResamplingTypes resampling=RSP_NearestNeighbor, esriRasterCompressionType compressionType=-1, long quality=75)  Line 1745 + 0x3b bytes	C++
 	GpDataManagementFunctions.dll!GPBuildPyramids::Execute(IArray * pParameters=0x0589db94, ITrackCancel * pTrackCancel=0x0589e07c, IGPEnvironmentManager * pEnvMgr=0x05891a74, IGPMessages * pMessages=0x0589b38c)  Line 246 + 0x43 bytes	C++

Change History (17)

comment:1 Changed 10 years ago by warmerdam

Keywords: overview added
Milestone: 1.6.4
Owner: changed from Warmerdam to warmerdam
Status: newassigned

A preliminary test with a 135000x135000 base file, made read-only and then running "gdaladdo big.tif 2" resulted in:

warmerda@gdal64[20]% ls -l big*
-r--r--r-- 1 warmerda warmerda  399788622 2010-02-01 23:34 big.tif
-rw-r--r-- 1 warmerda warmerda 4572057892 2010-02-01 23:51 big.tif.ovr

So creating a larger than 4GB file seems to have worked. This is with GDAL trunk, on linux.

I will try again tomorrow on windows. It takes quite a while!

comment:2 Changed 10 years ago by warmerdam

I have confirmed the problem does occur with the 1.6-esri branch on win32, but not with trunk on win32. I presume there has been some fixing related to forcing bigtiff mode that did not make it into 1.6-esri. I will investigate on that basis.

comment:3 Changed 10 years ago by warmerdam

I have ported some improvements to gt_overview.cpp from trunk into 1.6-esri (r18703). Now TIFF overviews should usually be done in BigTIFF is there is a significant chance of the .ovr file growing larger than 4GB.

So far, I'm not seeing problems with imagine format overviews on large files. It appears an .ige file is created. I will try a few more experiments.

comment:4 Changed 10 years ago by warmerdam

I have tried a few more experiments, and been unable to reproduce a crash with large .rrd files. Can you reproduce the problem with gdaladdo?

comment:5 Changed 10 years ago by gaopeng

OVR works for the original test case (uncompressed), but crashes for JPEG compression with quality 50.

RRD does not crash. But the resulting overiews are incorrect. For example, the generated overviews for a RGB image are grayscale.

comment:6 Changed 10 years ago by warmerdam

Peng Gao writes:

Here is what I found on creating large RRD, when testing a 79G IMG image.

  • The following condition (bCreateLargeRaster) is false so a separate .rrd is created, but it's too small, and overview generation failed.
        if( (psInfo->nEndOfFile 
             + (nOXSize * (double) nOYSize)
             * (HFAGetDataTypeBits(nDataType) / 8)) > 2000000000.0 )
            bCreateLargeRaster = TRUE;
    
        if( bCreateLargeRaster )
        {
            if( !HFACreateSpillStack( psInfo, nOXSize, nOYSize, 1, 
                                      64, nDataType, 
                                      &nValidFlagsOffset, &nDataOffset ) )
    	{
    	    return -1;
    	}
        }
    
  • Once the flag bCreateLargeRaster is forced to true, a small .rrd is created, but no .rde. The overviews are actually written to the existing .ige, which is not desirable.

The behavior I am looking for is:

  • If the uncompressed size of the entire overviews, including all bands, and all levels, is larger than 2G, create a large file.
  • Always create overviews in a separate .rrd (.rde) file.

comment:7 Changed 10 years ago by warmerdam

Gao,

I'm not really clear on why bCreateLargeRaster would have been determined to be false. A level 2x overview for a 79GB images should still be around 20GB.

I'm wondering if it is only some of the lower res overviews that are being treated as if they will fit in the .rrd file, but if the problem is that nEndOfFile does not reflect the size of previous overviews if they haven't been written yet. That is, if we create a 1.9GB overview, and a 500MB overview in quick succession before either are written it will not be obvious that cumulatively they are going to end up being more than 2GB. If this is the case the bug might only trigger in some fairly particular sizes. That might explain why I spent so much fruitless time trying to reproduce this bug.

comment:8 Changed 10 years ago by gaopeng

The first level overview created is at 4x, and the size calculation only uses 1 band. So the resulting size is 79G/3/16, which is about 1.7G.

I am not clear how nEndOfFile works. Is it for the base image or the ovr?

comment:9 Changed 10 years ago by warmerdam

Description: modified (diff)

OK, several problems now confirmed.

1) Creating jpeg compressed overviews on tiff with 1.6-esri does produce the expected error. Presumably the .ovr file is not created as big tiff when compression is used since it is hard to predict how large the file will get.

2) Creating .rrd files on a large file where each band overview is less than 2GB does not result in production of a spill file - at least for my test case. And the resulting >2GB file ends up having "2GB barrier wrap" issues so the imagery is corrupt - possibly the cause of apparent greyscale imagery?

Working on resolutions.

comment:10 Changed 10 years ago by warmerdam

Gao,

Do you use the BIGTIFF_OVERVIEW=IF_SAFER configuration option? I think this capability was implemented for you for this sort of situation. Basically it ensures that compressed overviews will be created as bigtiff in the .ovr file if the uncompressed size would be larger than 4GB. This should resolve the .ovr issue.

Let me know if I should test it - the run takes several hours on my system.

comment:11 Changed 10 years ago by gaopeng

The BIGTIFF_OVERVIEW=IF_SAFER configuration option was not set. I have just added the config option. But I was told people can't even create BigTIFF image anymore after the change.

I am looking at it, and will let you know if I find anything.

comment:12 Changed 10 years ago by warmerdam

OK, I've made significant progress on the HFA overview problems.

In trunk (r18859) I made changes to ensure that when .aux files are generated during overview creation we set the AUX=YES creation flag. This prevents the creation of a spill file for large images wasting as much space as needed for the uncompressed image.

In trunk (r18876) I:

1) made changes to hfaopen.cpp so that spill files for .rrd files are named .rde instead of .ige avoid conflict with base spill files in some cases.

2) made changes to hfaband.cpp so that the rule deciding on whether to create a spill file is based on the .rrd file, not the .aux file in cases where a .aux is involved. This fixes the "end of file will be past 2GB" computation when the overview is less than 2GB individually, but is cumulatively larger than 2GB.

All these changes have been backported to 1.7 (r18877) and 1.6-esri (r18878). I think this corrects all the HFA/rrd overview issues.

I have run a test with BIGTIFF_OVERVIEW IF_SAFER with jpeg overviews and was able to produce a .ovr file of 9GB. So that seems to be the key in that case.

I'll leave this ticket open till Gao can confirm the fixes.

comment:13 Changed 10 years ago by gaopeng

Frank,

Here is the feedback I got from testing.

Building rrd didn’t work right. The tool finished successfully but the layer properties state that pyramids don’t exist. I have two pyramid files: .rrd which is 1.72 GB in size, and a .rde file which is 407 MB in size. This took 2 hrs. I’ll start again to see if the behavior is consistent.

comment:14 Changed 10 years ago by gaopeng

OVR works with BIGTIFF_OVERVIEW IF_SAFER.

RRD works fine as well. The previous test was incorrect due to disk problem.

However, the overviews are split between two files, a .RRD (< 2G) and a .RDE. It works but is different from .IMG/.IGE pair, where .IMG is a reference, and all data reside in .IGE. It may not be an issue.

comment:15 Changed 10 years ago by warmerdam

Milestone: 1.7.2
Resolution: fixed
Status: assignedclosed

Gao,

In the case of .img/.ige the bands are all added in one go, and if they won't all fit in the .img they are all handled in the .ige. In the case of overviews, things are added one band/overview level at a time and things are added to the .img if they fit, otherwise they are handled via the spill file. I don't forsee any compatibility problems with this though it may seem a bit peculiar.

comment:16 Changed 10 years ago by gaopeng

Resolution: fixed
Status: closedreopened

The IMG file, regular (.img) or large (.ige) written after this fix can't be read in ERDAS IMAGINE.

comment:17 Changed 10 years ago by warmerdam

Resolution: fixed
Status: reopenedclosed

On review, I believe the problem was not introduced with the fix for this ticket, but instead as extra code in r18627 when passed on band descriptions during a createcopy. This results in empty layer names in the HFA file as per #3382.

I have fixed this in 1.6-esri (r19192).

Note: See TracTickets for help on using tickets.