Opened 14 years ago

Closed 12 years ago

#952 closed defect (worksforme)

Warper Array bounds error

Reported by: scott@… Owned by: warmerdam
Priority: high Milestone: 1.4.3
Component: GDAL_Raster Version: 1.3.0
Severity: major Keywords:
Cc: Mateusz Łoskot

Description (last modified by warmerdam)

Reprojecting SDTS DEM to WGS84 results in an array bounds error in GDALWarpNoDataMasker() in GDT_Float32 section when setting element of panValidityMask. panValidityMask is initialized in CreateKernelMask?() with poKernel->nBands (1) elements in BandSrcValid? section. GDALWarpNoDataMasker() is attempting to modify the 46444th element. nXSize = 1070. nYSize = 1389.

The data used can be made available in a 908 KB zip file.

GDAL 1.3.0

Attachments (2)

SDTS.zip (907.9 KB) - added by scott@… 14 years ago.
SDTS DEM
original-gdal13-gdalsvn.png (117.2 KB) - added by Mateusz Łoskot 13 years ago.
Visual review of (from left to right): original file - warped to WGS84 using GDAL 1.3.0 - warped to WGS84 using GDAL SVN (1.4.x)

Download all attachments as: .zip

Change History (13)

Changed 14 years ago by scott@…

Attachment: SDTS.zip added

SDTS DEM

comment:2 Changed 13 years ago by warmerdam

Cc: warmerdam added
Component: defaultGDAL_Raster
Description: modified (diff)
Milestone: 1.4.2
Owner: changed from warmerdam to Mateusz Łoskot
Version: unspecified1.3.0

Please see if you can reproduce this problem. Likely prudent to use valgrind to highlight any problems.

AGI is a Gold sponsor!

comment:3 Changed 13 years ago by Mateusz Łoskot

Status: newassigned

comment:4 Changed 13 years ago by Mateusz Łoskot

I think I'd need more details about exact steps to follow in order to reproduce the problem. Currently, I tried to use gdalwarp to transform the data to WGS84 and it seems to work well regarding the transformation:

  • Check format availability
D:\dev\gdal\bugs\952\sdts>gdalinfo --formats|grep SDTS
  SDTS (ro): SDTS Raster
  • Check the original data
D:\dev\gdal\bugs\952\sdts>gdalinfo -stats 1014CATD.DDF
GDAL: GDALOpen(1014CATD.DDF) succeeds as SDTS.
Driver: SDTS/SDTS Raster
Size is 1070, 1389
Coordinate System is:
PROJCS["UTM Zone 18, Northern Hemisphere",
    GEOGCS["NAD27",
        DATUM["North_American_Datum_1927",
            SPHEROID["Clarke 1866",6378206.4,294.978698213898,
                AUTHORITY["EPSG","7008"]],
            TOWGS84[-3,142,183,0,0,0,0],
            AUTHORITY["EPSG","6267"]],
        PRIMEM["Greenwich",0,
            AUTHORITY["EPSG","8901"]],
        UNIT["degree",0.0174532925199433,
            AUTHORITY["EPSG","9108"]],
        AXIS["Lat",NORTH],
        AXIS["Long",EAST],
        AUTHORITY["EPSG","4267"]],
    PROJECTION["Transverse_Mercator"],
    PARAMETER["latitude_of_origin",0],
    PARAMETER["central_meridian",-75],
    PARAMETER["scale_factor",0.9996],
    PARAMETER["false_easting",500000],
    PARAMETER["false_northing",0],
    UNIT["Meter",1]]
Origin = (478625.000000000000000,4427575.000000000000000)
Pixel Size = (10.000000000000000,-10.000000000000000)
Corner Coordinates:
Upper Left  (  478625.000, 4427575.000)
Lower Left  (  478625.000, 4413685.000)
Upper Right (  489325.000, 4427575.000)
Lower Right (  489325.000, 4413685.000)
Center      (  483975.000, 4420630.000)
Band 1 Block=1070x1 Type=Float32, ColorInterp=Undefined
  Minimum=-1.400, Maximum=96.100, Mean=15.882, StdDev=16.732
  NoData Value=-32766
  Unit Type: m
  Metadata:
    STATISTICS_MINIMUM=-1.3999999761581
    STATISTICS_MAXIMUM=96.099998474121
    STATISTICS_MEAN=15.882284699649
    STATISTICS_STDDEV=16.732221899913
GDAL: GDALClose(1014CATD.DDF)
GDAL: GDALDeregister_GTiff() called.
  • Transformation to WGS84 (Tiff output)
D:\dev\gdal\bugs\952\sdts>gdalwarp -t_srs EPSG:4326 1014CATD.DDF xCATD.DDF
GDAL: GDALOpen(1014CATD.DDF) succeeds as SDTS.
GDAL: GDALClose(1014CATD.DDF)
Creating output file that is 1243P x 1241L.
GDAL: GDALDriver::Create(GTiff,xCATD.DDF,1243,1241,1,Float32,00000000)
GDAL: GDALOpen(1014CATD.DDF) succeeds as SDTS.
Processing input file 1014CATD.DDF.
GDAL: GDALWarpKernel()::GWKNearestNoMasksFloat()
Src=0,0,1070x1389 Dst=0,0,1243x1241
:0...10...20...30...40...50...60...70...80...90...100 - done.
GDAL: GDALClose(1014CATD.DDF)
GDAL: GDALClose(xCATD.DDF)
GDAL: GDALDeregister_GTiff() called.
  • Check the output
D:\dev\gdal\bugs\952\sdts>gdalinfo -stats xCATD.DDF
GDAL: GDALOpen(xCATD.DDF) succeeds as GTiff.
Driver: GTiff/GeoTIFF
Size is 1243, 1241
Coordinate System is:
GEOGCS["WGS 84",
    DATUM["WGS_1984",
        SPHEROID["WGS 84",6378137,298.2572235630016,
            AUTHORITY["EPSG","7030"]],
        AUTHORITY["EPSG","6326"]],
    PRIMEM["Greenwich",0],
    UNIT["degree",0.0174532925199433],
    AUTHORITY["EPSG","4326"]]
Origin = (-75.250404119224768,40.000183183522374)
Pixel Size = (0.000101033361713,-0.000101033361713)
Metadata:
  AREA_OR_POINT=Area
Corner Coordinates:
Upper Left  ( -75.2504041,  40.0001832) ( 75d15'1.45"W, 40d 0'0.66"N)
Lower Left  ( -75.2504041,  39.8748008) ( 75d15'1.45"W, 39d52'29.28"N)
Upper Right ( -75.1248197,  40.0001832) ( 75d 7'29.35"W, 40d 0'0.66"N)
Lower Right ( -75.1248197,  39.8748008) ( 75d 7'29.35"W, 39d52'29.28"N)
Center      ( -75.1876119,  39.9374920) ( 75d11'15.40"W, 39d56'14.97"N)
Band 1 Block=1243x1 Type=Float32, ColorInterp=Gray
  Minimum=-32766.000, Maximum=96.100, Mean=-76.170, StdDev=1734.466
  Metadata:
    STATISTICS_MINIMUM=-32766
    STATISTICS_MAXIMUM=96.099998474121
    STATISTICS_MEAN=-76.169590166615
    STATISTICS_STDDEV=1734.4660008547
GDAL: GDALClose(xCATD.DDF)
GDAL: GDALDeregister_GTiff() called.

comment:5 Changed 13 years ago by warmerdam

Mateusz,

I would suggest you visually review the output file, looking for apparent data drop out. Also, try running the gdalwarp under valgrind to see if any array bounds errors pop out. I'd suggest also trying the same thing against gdal 1.3.0.

I suspect the problem is already fixed, but we want to verify.

comment:6 Changed 13 years ago by Mateusz Łoskot

  • gdalwarp 1.3.0 run under valgrind
mloskot:~/dev/gdal/bugs/952$ valgrind --leak-check=full ~/dev/gdal/release/gdal-1.3.0/apps/gdalwarp -t_srs EPSG:4326 1014CATD.DDF out.DDF
==30026== Memcheck, a memory error detector.
==30026== Copyright (C) 2002-2006, and GNU GPL'd, by Julian Seward et al.
==30026== Using LibVEX rev 1606, a library for dynamic binary translation.
==30026== Copyright (C) 2004-2006, and GNU GPL'd, by OpenWorks LLP.
==30026== Using valgrind-3.2.0-Debian, a dynamic binary instrumentation framework.
==30026== Copyright (C) 2000-2006, and GNU GPL'd, by Julian Seward et al.
==30026== For more details, rerun with: -v
==30026== 
GDAL: GDALOpen(1014CATD.DDF) succeeds as SDTS.

Creating output file that is 1243P x 1241L.
GDAL: GDALDriver::Create(GTiff,out.DDF,1243,1241,1,Float32,(nil))
GDAL: GDALWarpKernel()::GWKNearestNoMasksFloat()
Src=0,0,1070x1389 Dst=0,0,1243x1241
:0...10...20...30...40...50...60...70...80...90...100 - done.
GDAL: GDALClose(out.DDF)

GDAL: GDALClose(1014CATD.DDF)

GDAL: GDALDeregister_GTiff() called.
==30026== 
==30026== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 59 from 1)
==30026== malloc/free: in use at exit: 270,220 bytes in 34 blocks.
==30026== malloc/free: 10,861 allocs, 10,827 frees, 25,325,518 bytes allocated.
==30026== For counts of detected errors, rerun with: -v
==30026== searching for pointers to 34 not-freed blocks.
==30026== checked 1,555,304 bytes.
==30026== 
==30026== 1 bytes in 1 blocks are definitely lost in loss record 1 of 12
==30026==    at 0x4021396: malloc (vg_replace_malloc.c:149)
==30026==    by 0x487697F: strdup (in /lib/tls/i686/cmov/libc-2.4.so)
==30026==    by 0x420ECAC: VSIStrdup (in /home/mloskot/dev/gdal/release/gdal-1.3.0/libgdal.so)
==30026==    by 0x4205638: CPLStrdup (in /home/mloskot/dev/gdal/release/gdal-1.3.0/libgdal.so)
==30026==    by 0x40E857D: GTiffDataset::GTiffDataset() (in /home/mloskot/dev/gdal/release/gdal-1.3.0/libgdal.so)
==30026==    by 0x40ECF04: GTiffDataset::Create(char const*, int, int, int, GDALDataType, char**) (in /home/mloskot/dev/gdal/release/gdal-1.3.0/libgdal.so)
==30026==    by 0x41F5D95: GDALDriver::Create(char const*, int, int, int, GDALDataType, char**) (in /home/mloskot/dev/gdal/release/gdal-1.3.0/libgdal.so)
==30026==    by 0x41F6516: GDALCreate (in /home/mloskot/dev/gdal/release/gdal-1.3.0/libgdal.so)
==30026==    by 0x804AB69: main (in /home/mloskot/dev/gdal/release/gdal-1.3.0/apps/gdalwarp)
==30026== 
==30026== LEAK SUMMARY:
==30026==    definitely lost: 1 bytes in 1 blocks.
==30026==      possibly lost: 0 bytes in 0 blocks.
==30026==    still reachable: 270,219 bytes in 33 blocks.
==30026==         suppressed: 0 bytes in 0 blocks.

Memory leak exist, one byte is lost.

comment:7 Changed 13 years ago by Mateusz Łoskot

  • gdalwarp (SVN) under valgrind
mloskot:~/dev/gdal/bugs/952$ valgrind --leak-check=full ~/dev/gdal/_svn/trunk/gdal/apps/gdalwarp -t_srs EPSG:4326 1014CATD.DDF out1.DDF
...
==30028== 
Creating output file that is 1243P x 1241L.
Processing input file 1014CATD.DDF.
:0...10...20...30...40...50...60...70...80...90...100 - done.
==30028== Invalid free() / delete / delete[]
==30028==    at 0x4020FB0: free (vg_replace_malloc.c:233)
==30028==    by 0x67A864D: (within /lib/tls/i686/cmov/libc-2.4.so)
==30028==    by 0x67A8256: __libc_freeres (in /lib/tls/i686/cmov/libc-2.4.so)
==30028==    by 0x401D1F6: _vgnU_freeres (vg_preloaded.c:60)
==30028==    by 0x672C1F3: _Exit (in /lib/tls/i686/cmov/libc-2.4.so)
==30028==    by 0x66B28D3: (below main) (in /lib/tls/i686/cmov/libc-2.4.so)
==30028==  Address 0x6583CC8 is not stack'd, malloc'd or (recently) free'd
==30028== 
==30028== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 141 from 1)
==30028== malloc/free: in use at exit: 265,738 bytes in 34 blocks.
==30028== malloc/free: 13,384 allocs, 13,351 frees, 25,252,279 bytes allocated.
==30028== For counts of detected errors, rerun with: -v
==30028== searching for pointers to 34 not-freed blocks.
==30028== checked 4,561,268 bytes.
==30028== 
==30028== LEAK SUMMARY:
==30028==    definitely lost: 0 bytes in 0 blocks.
==30028==      possibly lost: 0 bytes in 0 blocks.
==30028==    still reachable: 265,738 bytes in 34 blocks.
==30028==         suppressed: 0 bytes in 0 blocks.

No memory leak detected.

In both cases no issue like out of range of array bound has been detected.

Changed 13 years ago by Mateusz Łoskot

Attachment: original-gdal13-gdalsvn.png added

Visual review of (from left to right): original file - warped to WGS84 using GDAL 1.3.0 - warped to WGS84 using GDAL SVN (1.4.x)

comment:8 Changed 13 years ago by Mateusz Łoskot

Cc: Mateusz Łoskot added; warmerdam removed
Owner: changed from Mateusz Łoskot to warmerdam
Status: assignednew

comment:9 Changed 13 years ago by warmerdam

Summary: Array bounds errorWarper Array bounds error

comment:10 Changed 12 years ago by warmerdam

Scott,

I think we need more information to reproduce this. We don't seem to see any obvious problem at our end.

comment:11 Changed 12 years ago by warmerdam

Milestone: 1.4.21.4.3

Putting off to 1.4.3 since we aren't able to reproduce it yet.

comment:12 Changed 12 years ago by warmerdam

Resolution: worksforme
Status: newclosed

Unable to reproduce, and no further feedback so I'm closing this one.

Note: See TracTickets for help on using tickets.