HDF4 create test crashes for int32 & uint32, OSX 64bit

[kyngchaos]

As title says. It looks like it's happening when the dataset is closed. From the autotest run:

  TEST: Create: int32.tif rank=3 ... python2.6(60497,0x7fff7101abe0) malloc: *** error for object 
0x1009a1e08: incorrect checksum for freed object - object was probably modified after being freed.

and the OSX crash report:

Exception Type:  EXC_CRASH (SIGABRT)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Crashed Thread:  0  Dispatch queue: com.apple.main-thread

Application Specific Information:
*** error for object 0x1009a1e08: incorrect checksum for freed object - object was probably modified after being freed.

Thread 0 Crashed:  Dispatch queue: com.apple.main-thread
0   libSystem.B.dylib             	0x00007fff884e8fe6 __kill + 10
1   libSystem.B.dylib             	0x00007fff88589e32 abort + 83
2   libSystem.B.dylib             	0x00007fff88578ae5 szone_error + 519
3   libSystem.B.dylib             	0x00007fff884a562f small_free_list_remove_ptr + 154
4   libSystem.B.dylib             	0x00007fff884a212f szone_free_definite_size + 3254
5   org.gdal.gdal                 	0x0000000101621136 sd_NC_free_cdf + 50
6   org.gdal.gdal                 	0x0000000101623a70 sd_ncclose + 232
7   org.gdal.gdal                 	0x00000001016293a2 SDend + 200
8   org.gdal.gdal                 	0x00000001010c58ce HDF4ImageDataset::~HDF4ImageDataset() + 66
9   org.gdal.gdal                 	0x00000001011b66ea GDALClose + 152
10  _gdal.so                      	0x00000001005010e5 _wrap_delete_Dataset + 306
11  org.python.python             	0x0000000100017173 PyObject_Call + 112
12  org.python.python             	0x0000000100017367 PyObject_CallFunctionObjArgs + 215
13  _gdal.so                      	0x00000001004fa2b3 SwigPyObject_dealloc + 117
14  org.python.python             	0x0000000100044fb4 PyDict_Contains + 2517

Same HDF4.2r4 binaries that worked in GDAL 1.6. All other HDF4 tests with int32 and uint32 succeed (I blocked out the failures in hdf4_write.py so it would continue with the tests). I couldn't figure out how to emulate this test outside of Python to see if it was the python interface that's the problem.

[kyngchaos]

P.S. Implied, but not stated above: it works on OSX 32bit.

[warmerdam]

I have run autotest/gcore/hfa_write.py under valgrind on 64bit linux with nothing showing up. Without access to 64bit macos x there isn't much more that I can do.

[rouault]

William,

are you 200% positive that you have linked against HDF4.2r2 ? Because, on my 64bit Linux, when linking against an older HDF4 (4.1r4), I got the following crash :

TEST: Create: int32.tif rank=3 ... *** glibc detected *** /usr/bin/python: double free or corruption (out): 0x00000000008fe370 ***

and Valgrind reported :

==13402== Invalid write of size 1
==13402==    at 0x4C250AF: memcpy (mc_replace_strmem.c:402)
==13402==    by 0x8AF173F: DFKnb4b (in /usr/lib/libdf.so.4.1r4)
==13402==    by 0x88913B2: (within /usr/lib/libmfhdf.so.4.1r4)
==13402==    by 0x88923BE: NCvario (in /usr/lib/libmfhdf.so.4.1r4)
==13402==    by 0x889A41A: SDreaddata (in /usr/lib/libmfhdf.so.4.1r4)
==13402==    by 0x631E85E: HDF4ImageRasterBand::IReadBlock(int, int, void*) (hdf4imagedataset.cpp:303)
==13402==    by 0x64DAD89: GDALRasterBand::GetLockedBlockRef(int, int, int) (gdalrasterband.cpp:1249)
==13402==    by 0x64E9A19: GDALRasterBand::IRasterIO(GDALRWFlag, int, int, int, int, void*, int, int, GDALDataType, int, int) (rasterio.cpp:100)
==13402==    by 0x65129F2: GDALChecksumImage (gdalchecksum.cpp:147)
==13402==    by 0x5E498CA: _wrap_Band_Checksum (gdal_wrap.cpp:3792)
==13402==    by 0x417E32: PyObject_Call (in /usr/bin/python2.5)
==13402==    by 0x487072: PyEval_EvalFrameEx (in /usr/bin/python2.5)
==13402==  Address 0x5d5ea27 is 3 bytes after a block of size 4 free'd
==13402==    at 0x4C22B2E: free (vg_replace_malloc.c:323)
==13402==    by 0x88903E4: SDPfreebuf (in /usr/lib/libmfhdf.so.4.1r4)
==13402==    by 0x88911FA: (within /usr/lib/libmfhdf.so.4.1r4)
==13402==    by 0x88923BE: NCvario (in /usr/lib/libmfhdf.so.4.1r4)
==13402==    by 0x889A41A: SDreaddata (in /usr/lib/libmfhdf.so.4.1r4)
==13402==    by 0x631E85E: HDF4ImageRasterBand::IReadBlock(int, int, void*) (hdf4imagedataset.cpp:303)
==13402==    by 0x64DAD89: GDALRasterBand::GetLockedBlockRef(int, int, int) (gdalrasterband.cpp:1249)
==13402==    by 0x64E9A19: GDALRasterBand::IRasterIO(GDALRWFlag, int, int, int, int, void*, int, int, GDALDataType, int, int) (rasterio.cpp:100)
==13402==    by 0x65129F2: GDALChecksumImage (gdalchecksum.cpp:147)
==13402==    by 0x5E498CA: _wrap_Band_Checksum (gdal_wrap.cpp:3792)
==13402==    by 0x417E32: PyObject_Call (in /usr/bin/python2.5)
==13402==    by 0x487072: PyEval_EvalFrameEx (in /usr/bin/python2.5)

When upgrading to HDF4.2r2, no problem any more.

[kyngchaos]

I'm positive it's 4.2r4. No other 4.x version installed.

I could rebuild on OSX 10.5 to see what valgrind has to say (valgrind does not run on 10.6 yet), if that'll help.

[kyngchaos]

With CPL_DEBUG on, and some debug print statements added to gdaltest.py, I see that strangely it's not happening when closing the created dataset, but after reopening it, when checking the checksum (maybe that is closing the bands?):

GDAL: GDALOpen(tmp/int32.tif.tst, this=0x104023e00) succeeds as HDF4.
checksuming band 1
python2.6(86017,0x7fff7101abe0) malloc: *** error for object 0x104057c08: incorrect checksum for freed 
object - object was probably modified after being freed.

And also strange, just before this I had run the int32 test without it crashing, and it did report the checksum failed. It will only crash now.

[kyngchaos]

OK, can't check valgrind - [u]int32 tests succeed on OSX 10.5 64bit...

Small changes in the source could have triggered an optimization bug in GCC 4.2, it's happened before.

I compiled my HDF4 library last Sep, at the first release of OSX 10.6 Xcode 3.2, and I'm at 3.2.1 (I think GCC was 4.2.1 from the start, just a different build). I can try recompiling HDF4.

[kyngchaos]

optimization isn't the problem. With unoptimized HDF4 and GDAL, same crash. But there are more details in the trace:

0   libSystem.B.dylib             	0x00007fff884e8fe6 __kill + 10
1   libSystem.B.dylib             	0x00007fff88589e32 abort + 83
2   libSystem.B.dylib             	0x00007fff88578ae5 szone_error + 519
3   libSystem.B.dylib             	0x00007fff884a568b small_free_list_remove_ptr + 246
4   libSystem.B.dylib             	0x00007fff884a212f szone_free_definite_size + 3254
5   org.gdal.gdal                 	0x0000000101869cc7 free_biobuf + 28
6   org.gdal.gdal                 	0x000000010186a532 xdrposix_destroy + 85
7   org.gdal.gdal                 	0x000000010184be7b sd_NC_free_cdf + 112
8   org.gdal.gdal                 	0x000000010185378e sd_ncclose + 295
9   org.gdal.gdal                 	0x0000000101857ab4 SDend + 299
10  org.gdal.gdal                 	0x0000000101141ab7 HDF4ImageDataset::~HDF4ImageDataset() + 97
11  org.gdal.gdal                 	0x00000001012bfcd0 GDALClose + 234
12  _gdal.so                      	0x00000001004fbb5a delete_GDALDatasetShadow(void*) + 39
13  _gdal.so                      	0x0000000100511457 _wrap_delete_Dataset + 197

I do remember that figuring out how to fix the XDR stuff in HDF4 for OSX was a PITA, and I may have missed something that only showed up now.

Maybe it's something in WriteRaster? that writes bad data which causes the checksum to crash? I see that the Create test also copies but as a separate step, with WriteRaster?.

[kyngchaos]

With CPL_DEBUG on so I could save the test files, I compared the successful test with the failure. It looks like the failed HDF output reports as a 3-band image, though looking at it with a hex editor I see only a single bands' worth of data. The data is exactly the same as the correct HDF file (I did find and fix one 64bit issue I missed in HDF4 lib). But there is a bunch of extra junk in the trailer (?). And gdalinfo -checksum crashes on it.

[kyngchaos]

failed int32 (rank=3) test output