Opened 19 years ago

Last modified 19 years ago

#774 closed defect (fixed)

Large IMG files (over 6GB) problems

Reported by: bolsen@… Owned by: warmerdam
Priority: high Milestone:
Component: default Version: unspecified
Severity: normal Keywords: VERIFIED


Tried to cut 1GB geotiff chunks out of an 11GB Erdas img file.

Translate would abort when accessing one of the offset bands.

There's an overflow problem not being properly handled in hfaband.cpp and

The short of it is that the img field:  "layerStackDataOffset[1]" is used for
64bit file offset.  Currently it's being hard set and read with value '0'.

Tried today's build, same problem (cvs 20070217)

I can verify that these suggested fixed are NOT cross platform, in fact they
break on amd64.

Suggested fixes:

hfaband.cpp:  HFABand::LoadExternalBlockInfo()
roughly line 480:

    nBlockStart = poDMS->GetIntField( "layerStackDataOffset[0]" );

    unsigned long low
       = (unsigned long)poDMS->GetIntField( "layerStackDataOffset[0]" );
    unsigned long high
       = (unsigned long)poDMS->GetIntField( "layerStackDataOffset[1]" );

    unsigned long* sPtr = (unsigned long*)&nBlockStart;
    *sPtr = low;
    *sPtr = high;

hfaopen.cpp: HFACreate
about line 1741

   vsi_l_offset blockOffset = (vsi_l_offset)(iHeaderSize)
      + (vsi_l_offset)(nBands) * (vsi_l_offset)(iFlagsSize);

   poEdms_State->SetIntField( "layerStackDataOffset[0]" , (int) (blockOffset >>
32) );
   poEdms_State->SetIntField( "layerStackDataOffset[1]" , (int) (blockOffset &
0xffffffff) );

       poEdms_State->SetIntField( "layerStackDataOffset[0]",
                   iHeaderSize + nBands * iFlagsSize );
       poEdms_State->SetIntField( "layerStackDataOffset[1]", 0 );

Attachments (2)

d_Imagine_L2_1503RGBF16L2.img (7.5 KB ) - added by bolsen@… 19 years ago.
IMG file which accompanies this 11G 3 band IGE file
d_L2_1608RGBF16L2.img (7.5 KB ) - added by bolsen@… 19 years ago.
another test case, my fixes failed to extract band 3 (blue)

Download all attachments as: .zip

Change History (7)

comment:1 by warmerdam, 19 years ago


Could you provide the .img file in question?  I don't need the .ige, just
the .img should be fine.  I think I have been using just the low order 32bits
of the offset in the past and it worked since I was working with one band
files where the offset would a relatively small value. 

I plan to apply a patch roughly like yours, but using the GBigInt type instead
of long long, and just regular integer arithmetic instead of your way of
composing the low and high words. 

comment:2 by bolsen@…, 19 years ago

Let me try to attach the img file here...

by bolsen@…, 19 years ago

IMG file which accompanies this 11G 3 band IGE file

comment:3 by bolsen@…, 19 years ago

Oh, if it would be useful I could also have the guy generate a 5 band 16bit img
file (RGBPanIR).  I think  we could get it in the 30+GB range with one of those.

by bolsen@…, 19 years ago

Attachment: d_L2_1608RGBF16L2.img added

another test case, my fixes failed to extract band 3 (blue)

comment:4 by warmerdam, 19 years ago


OK, I have committed fixes for this.  I moved stuff into a GetBigIntField()
method on the HFAEntry and used up shifting of the high order 32bit word
which I hope will work properly on big or little endian systems. 

I would appreciate it if you could try this some time to verify my fix.  My
check was really to see I was getting the right looking values for offsets,
but without the real ige file I can't easily confirm complete correct operation.

I also looked into the creation code, but in that case we always create one
"layer" with multiple bands, instead of the approach that imagine seems
to be using of creating many layers each with one band.  If we ever change
the GDAL creation strategy for spill files, we will need similar logic on
the write side. 

Thanks for the report, and sample .img files!

comment:5 by bolsen@…, 19 years ago

Tested on amd64: fedora core 3, fedora core 2, i386: redhat 7.3, fedora core 3 
Seems to read files properly.
Note: See TracTickets for help on using tickets.