Integer overflows in rasterfill.cpp

[akhliustov]

1. http://trac.osgeo.org/gdal/browser/trunk/gdal/alg/rasterfill.cpp#L571
   
   

if( iY - panLastY[iX] <= dfMaxSearchDist )

panLastY[iX] is unsigned, so iY - panLastY[iX] is unsigned too. If iY < panLastY[iX], their "difference" is a large positive number that can be greater than dfMaxSearchDist. In that case the condition evaluates to false, which is (probably) not expected.

2. http://trac.osgeo.org/gdal/browser/trunk/gdal/alg/rasterfill.cpp#L333
   
   double dfDistSq = ((target_x-origin_x) * (target_x-origin_x)) + ((target_y-origin_y) * (target_y-origin_y));
   
   The right-hand expression here can have overflow on small enough input (I will try to attach it ASAP).

[akhliustov]

Here is the raster (.in.tif) with 800 rows and 1200 columns. Running
gdal_fillnodata -co COMPRESS=LZW -mask fillnodata.in.tif -md 0 fillnodata.in.tif fillnodata.out.tif
produces an output (.out.tif) which has up to 10 rows of garbage (the topmost ones).
In my program I use the function directly but the result is the same.

[Even Rouault]

Thanks!

trunk r26340, branches/1.10 r26341 "GDALFillNodata(): Fix use of uninitialized memory and integer overflows (#4010, #5203)"