x64 issue in BT dataset driver

[murphke1]

Found what appears to be a GDAL bug today. Attempting to use "gdal_translate" on a 4 Gig BT file. Note that my GDAL code date is 20120516, in case its already been addressed.

In "btdataset.cpp" the line below fails because the conversion to int64 doesn't happen until after the multiply and add. None of the multipliers are int64 so you get integer overflow before the value is converted to int64 when it gets passed in to the function.

/* -------------------------------------------------------------------- */
/*      Seek to profile.                                                */
/* -------------------------------------------------------------------- */
   if( VSIFSeekL( fpImage, 
                  256 + nBlockXOff * nDataSize * nRasterYSize, 
                  SEEK_SET ) != 0 )	
   {
       CPLError( CE_Failure, CPLE_FileIO, 
                 ".bt Seek failed:%s", VSIStrerror( errno ) );
       return CE_Failure;
   }

Replacing it with the below sorted that out. Still trying to process the file - there may be other int64 issues.

/* -------------------------------------------------------------------- */
/*      Seek to profile.                                                */
/* -------------------------------------------------------------------- */
   if( VSIFSeekL( fpImage, 
                  256 + (__int64)nBlockXOff * (__int64)nDataSize * (__int64)nRasterYSize, 
                  SEEK_SET ) != 0 )
   {
       CPLError( CE_Failure, CPLE_FileIO, 
                 ".bt Seek failed:%s", VSIStrerror( errno ) );
       return CE_Failure;
   }

Kevin Murphy

[warmerdam]

I have applied a slight variation on this change in trunk (r24724). Could you confirm things work with this change? Once confirmed the change could be backported to 1.9 and the ticket closed.

[murphke1]

It looks like it ought to work fine. I'll try to recompile and check it out tomorrow.

[murphke1]

Works fine for me with your version of the code.

[warmerdam]

Back ported change to 1.9 (r24733).