Large IMG files (over 6GB) problems

[bolsen@…]

Tried to cut 1GB geotiff chunks out of an 11GB Erdas img file.

Translate would abort when accessing one of the offset bands.

There's an overflow problem not being properly handled in hfaband.cpp and
hfaopen.cpp.

The short of it is that the img field:  "layerStackDataOffset[1]" is used for
64bit file offset.  Currently it's being hard set and read with value '0'.

Tried today's build, same problem (cvs 20070217)

I can verify that these suggested fixed are NOT cross platform, in fact they
break on amd64.

Suggested fixes:

hfaband.cpp:  HFABand::LoadExternalBlockInfo()
roughly line 480:

   /*
    nBlockStart = poDMS->GetIntField( "layerStackDataOffset[0]" );
   */

    unsigned long low
       = (unsigned long)poDMS->GetIntField( "layerStackDataOffset[0]" );
    unsigned long high
       = (unsigned long)poDMS->GetIntField( "layerStackDataOffset[1]" );

    unsigned long* sPtr = (unsigned long*)&nBlockStart;
    *sPtr = low;
    ++sPtr;
    *sPtr = high;


hfaopen.cpp: HFACreate
about line 1741

   vsi_l_offset blockOffset = (vsi_l_offset)(iHeaderSize)
      + (vsi_l_offset)(nBands) * (vsi_l_offset)(iFlagsSize);

   poEdms_State->SetIntField( "layerStackDataOffset[0]" , (int) (blockOffset >>
32) );
   poEdms_State->SetIntField( "layerStackDataOffset[1]" , (int) (blockOffset &
0xffffffff) );

/*
       poEdms_State->SetIntField( "layerStackDataOffset[0]",
                   iHeaderSize + nBands * iFlagsSize );
       poEdms_State->SetIntField( "layerStackDataOffset[1]", 0 );
*/

[warmerdam]

Brian,

Could you provide the .img file in question?  I don't need the .ige, just
the .img should be fine.  I think I have been using just the low order 32bits
of the offset in the past and it worked since I was working with one band
files where the offset would a relatively small value. 

I plan to apply a patch roughly like yours, but using the GBigInt type instead
of long long, and just regular integer arithmetic instead of your way of
composing the low and high words. 

[bolsen@…]

Let me try to attach the img file here...

[bolsen@…]

IMG file which accompanies this 11G 3 band IGE file

[bolsen@…]

Oh, if it would be useful I could also have the guy generate a 5 band 16bit img
file (RGBPanIR).  I think  we could get it in the 30+GB range with one of those.

[bolsen@…]

another test case, my fixes failed to extract band 3 (blue)

[warmerdam]

Brian,

OK, I have committed fixes for this.  I moved stuff into a GetBigIntField()
method on the HFAEntry and used up shifting of the high order 32bit word
which I hope will work properly on big or little endian systems. 

I would appreciate it if you could try this some time to verify my fix.  My
check was really to see I was getting the right looking values for offsets,
but without the real ige file I can't easily confirm complete correct operation.

I also looked into the creation code, but in that case we always create one
"layer" with multiple bands, instead of the approach that imagine seems
to be using of creating many layers each with one band.  If we ever change
the GDAL creation strategy for spill files, we will need similar logic on
the write side. 

Thanks for the report, and sample .img files!

[bolsen@…]

Tested on amd64: fedora core 3, fedora core 2, i386: redhat 7.3, fedora core 3 
Seems to read files properly.