Creating 2-4GB Imagine Files Broken?

[warmerdam]

This bug was reported by Kathyrn (Yao) Peng from PCI Geomatics.
(peng a t pcigeomatics dot com):


Hi Frank,
My name is Kathryn, I work for PCIGeomatics group.
Recently I upgrade our Erdas Imagine (.img) format.
When we test it we found the problem below.

When I translate the fake file to .img  it goes through the motions.
but nothing get created if it is bigger than 2 gigs.
And we do not get any warnings that nothing will be created.

when I debug through I found:
1) In function hfaband.cpp=>LoadBlockInfo

    for( iBlock = 0; iBlock < nBlocks; iBlock++ )
    {
        char	szVarName[64];
        int	nLogvalid, nCompressType;
	
	  sprintf( szVarName, "blockinfo[%d].offset", iBlock );
        panBlockStart[iBlock] = poDMS->GetIntField( szVarName );
        
        ...
    }

When current channel is 3 and iBlock reach to number 82110,
panBlockStart[iBlock] get a huge strange number which is wrong.

2) At this point I lloked in hfafield=>ExtractInstValue

	switch( chItemType )
	{
		....
		case 'L':
      	{
          		GInt32	nNumber;
          
          		memcpy( &nNumber, pabyData + nIndexValue*4, 4 );
          		HFAStandard( 4, &nNumber );
          		nIntRet = nNumber;
          		dfDoubleRet = nIntRet;
      	}

		...
	}
I found for some reason "memcpy( &nNumber, pabyData + nIndexValue*4, 4
)"
will give us a big negative number, this will cause a wrong
panBlockStart(1).

3)Later when we call 
	
GDBByteChanIO=>...=>EIIReadBlock=>...=>GetRasterBlock=>VSIFReadL, 

we use wrong block offset in DKFRead, this cause program failed.

I made a temporary fix by changing line "nIntRet = nNumber;" to
"nIntRet = ABS( nNumber );" to change the negative number to positive.
This works fine for current case.

I reprot this to you because I think you may want to take a look and
make
a better fix from overall view of this format.

The attached file is the fake format data set we used.
If you need more information, please contact me at yao@pcigeomatics.com.

best regard,
Kathryn

[warmerdam]

Note: this is CSR 17568 at PCI.

The "fake" file provided appears to be a special PCI format for test files
representing a 30000 x 30000 x 3 bands file of type 8U. This would be about
2.7 GB.  So it should not need to use the spill file, but some offsets
will be over the size representable in a signed integer. 

The code in GetIntField() will indeed return block offsets in a signed 
integer, and value over the 2GB mark will be returned as a negative number. 

...

Gar, I see rev 1.34 of hfaband.cpp has:

 * $Log: hfaband.cpp,v $
 * Revision 1.34  2003/07/29 10:09:43  dron
 * Fixed problem with creation of ordinary files larger 2GB.

It changes the poDMS->GetIntField() call to be cast to unsigned before
assigning to panBlockStart():

        sprintf( szVarName, "blockinfo[%d].offset", iBlock );
        panBlockStart[iBlock] = (GUInt32)poDMS->GetIntField( szVarName );

This is the correct fix.