Opened 4 years ago

Closed 4 years ago

#5203 closed defect (fixed)

Integer overflows in rasterfill.cpp

Reported by: akhliustov Owned by: warmerdam
Priority: normal Milestone: 1.10.1
Component: Algorithms Version: unspecified
Severity: normal Keywords:
Cc:

Description (last modified by akhliustov)

  1. http://trac.osgeo.org/gdal/browser/trunk/gdal/alg/rasterfill.cpp#L571

if( iY - panLastY[iX] <= dfMaxSearchDist )

panLastY[iX] is unsigned, so iY - panLastY[iX] is unsigned too. If iY < panLastY[iX], their "difference" is a large positive number that can be greater than dfMaxSearchDist. In that case the condition evaluates to false, which is (probably) not expected.

  1. http://trac.osgeo.org/gdal/browser/trunk/gdal/alg/rasterfill.cpp#L333

    double dfDistSq = ((target_x-origin_x) * (target_x-origin_x)) + ((target_y-origin_y) * (target_y-origin_y));

    The right-hand expression here can have overflow on small enough input (I will try to attach it ASAP).

Attachments (3)

rasterfill.unsigned.patch (1004 bytes) - added by akhliustov 4 years ago.
fillnodata.in.tif (135.5 KB) - added by akhliustov 4 years ago.
fillnodata.out.tif (374.3 KB) - added by akhliustov 4 years ago.

Download all attachments as: .zip

Change History (6)

Changed 4 years ago by akhliustov

Attachment: rasterfill.unsigned.patch added

comment:1 Changed 4 years ago by akhliustov

Description: modified (diff)
Summary: Implicit signed-to-unsigned conversions in rasterfill.cppInteger overflows in rasterfill.cpp

Changed 4 years ago by akhliustov

Attachment: fillnodata.in.tif added

Changed 4 years ago by akhliustov

Attachment: fillnodata.out.tif added

comment:2 Changed 4 years ago by akhliustov

Here is the raster (.in.tif) with 800 rows and 1200 columns. Running
gdal_fillnodata -co COMPRESS=LZW -mask fillnodata.in.tif -md 0 fillnodata.in.tif fillnodata.out.tif
produces an output (.out.tif) which has up to 10 rows of garbage (the topmost ones).
In my program I use the function directly but the result is the same.

comment:3 Changed 4 years ago by Even Rouault

Milestone: 1.10.1
Resolution: fixed
Status: newclosed

Thanks!

trunk r26340, branches/1.10 r26341 "GDALFillNodata(): Fix use of uninitialized memory and integer overflows (#4010, #5203)"

Note: See TracTickets for help on using tickets.