#6448 closed defect (fixed)

gdalwarp failed

Reported by: liminlu0314 Owned by: warmerdam
Priority: normal Milestone: 1.11.5
Component: Algorithms Version:
Severity: normal Keywords: gdalwarp, int overflow
Cc:

Description (last modified by liminlu0314)

When the value of GDALWarpOptions::dfWarpMemoryLimit is set to be very large, such as 10 GB, the program will fail, because the size of allocated memory overflows. The original code is as below:

gdalwarpoperation.cpp line 2059

    if( *ppMask == NULL )
    {
        int nBytes; //this line should be GIntBig nBytes;

        if( nBitsPerPixel == 32 )
            nBytes = (nXSize * nYSize + nExtraElts) * 4;
        else
            nBytes = (nXSize * nYSize + nExtraElts + 31) / 8;

        *ppMask = VSI_MALLOC_VERBOSE( nBytes );

        if( *ppMask == NULL )
        {
            return CE_Failure;
        }

        memset( *ppMask, nDefault, nBytes );
    }

The modified code is as follows:

    if( *ppMask == NULL )
    {
        GIntBig  nBytes; 

        if( nBitsPerPixel == 32 )
            nBytes = ((GIntBig )nXSize * nYSize + nExtraElts) * 4;
        else
            nBytes = ((GIntBig )nXSize * nYSize + nExtraElts + 31) / 8;

        *ppMask = VSI_MALLOC_VERBOSE( nBytes );

        if( *ppMask == NULL )
        {
            return CE_Failure;
        }

        memset( *ppMask, nDefault, nBytes );
    }

Attachments (1)

gdal_bug.jpg (301.8 KB) - added by liminlu0314 13 months ago.

Download all attachments as: .zip

Change History (3)

Changed 13 months ago by liminlu0314

Attachment: gdal_bug.jpg added

comment:1 Changed 13 months ago by liminlu0314

Description: modified (diff)

comment:2 Changed 13 months ago by Even Rouault

Resolution: fixed
Status: newclosed
Version: svn-trunk

trunk r33918, branches/2.0 r33919, branches/1.11 r33920 "GDALWarpOperation::CreateKernelMask?(): fix potential 32 bit integer overflow when using warp memory value > 2GB (derived from patch by liminlu0314, #6448)"

Note: See TracTickets for help on using tickets.