Integer overflow and EXC_BAD_INSTRUCTION in gdalwarpkernel.cpp with Apple LLVM version 8.0.0

[sgillies]

There's a check for integer overflow after line 3956 in gdalwarpkernel.cpp, but it can come too late if we've overflowed before the subtraction at line 3953.

* thread #1: tid = 0x6246ea, 0x000000010498a073 libgdal.20.dylib`GWKCheckAndComputeSrcOffsets(_pabSuccess=0x000000010bf51990, _iDstX=0, _padfX=0x000000010ed96a70, _padfY=0x0000000113a48550, _poWK=0x00007fff5bff1378, _nSrcXSize=490, _nSrcYSize=1, iSrcOffset=0x00007fff5bff0e44) + 275 at gdalwarpkernel.cpp:3953, queue = 'com.apple.main-thread', stop reason = EXC_BAD_INSTRUCTION (code=EXC_I386_INVOP, subcode=0x0)
    frame #0: 0x000000010498a073 libgdal.20.dylib`GWKCheckAndComputeSrcOffsets(_pabSuccess=0x000000010bf51990, _iDstX=0, _padfX=0x000000010ed96a70, _padfY=0x0000000113a48550, _poWK=0x00007fff5bff1378, _nSrcXSize=490, _nSrcYSize=1, iSrcOffset=0x00007fff5bff0e44) + 275 at gdalwarpkernel.cpp:3953
   3950	    int iSrcX, iSrcY;
   3951
   3952	    iSrcX = ((int) (_padfX[_iDstX] + 1e-10)) - _poWK->nSrcXOff;
-> 3953	    iSrcY = ((int) (_padfY[_iDstX] + 1e-10)) - _poWK->nSrcYOff;
   3954
   3955	    /* If operating outside natural projection area, padfX/Y can be */
   3956	    /* a very huge positive number, that becomes -2147483648 in the */
(lldb) p _padfY[_iDstX] + 1e-10
(double) $2 = 3170313390.4902153
(lldb) p (int) _padfY[_iDstX] + 1e-10
(double) $3 = -2147483648

My compiler details:

$ gcc --version
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 8.0.0 (clang-800.0.42.1)
Target: x86_64-apple-darwin16.1.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

I don't see this on Linux and didn't see it before upgrading my OS (with new clang). Could be specific to clang-800?

[Even Rouault]

In 36994:

Warper: avoid undefined behaviour when doing floating point to int conversion, that may trigger exception with some compilers (LLVM 8) (fixes #6753)

[Even Rouault]

In 36995:

Warper: avoid undefined behaviour when doing floating point to int conversion, that may trigger exception with some compilers (LLVM 8) (fixes #6753)

[Even Rouault]

Sean, could you test if the above fix works ? I guess LLVM8 enables -fsanitize=undefined, or a subset of it, by default.

[sgillies]

I have confirmed the fix. Thanks, Even!