64bit linux port: fdocore unit test failures

[warmerdam]

Getting the following failures in the Fdo core unit tests on 64bit linux (AMD64 / Ubuntu):

(first failure actually reflects a few tweaks already in FilterParseTest)

FilterParseTest.cpp:312:Assertion Test name: 15FilterParseTest.testBoundaryValues pLLVal2 != NULL

ExpressionParseTest.cpp:93:Assertion Test name: 19ExpressionParseTest.testExpressions Parse/ToString do not match!

<1567312775> should be <12345678901234567>

ExpressionParseTest.cpp:402:Assertion Test name: 19ExpressionParseTest.testBoundaryValues pLVal2->GetInt32() == LONG_MIN

UnitTestUtil.cpp:47:Assertion Test name: 7XmlTest.testCopy FDO_45_ITEMINCOLLECTION

UnitTestUtil.cpp:47:Assertion Test name: 18SpatialContextTest.testXmlNesting SC_4_XMLNOREAD

UnitTestUtil.cpp:47:Assertion Test name: 10SchemaTest.testXmlV2 FDO_45_ITEMINCOLLECTION

UnitTestUtil.cpp:47:Assertion Test name: 10SchemaTest.testXmlAll FDO_45_ITEMINCOLLECTION

UnitTestUtil.cpp:47:Assertion Test name: 10SchemaTest.testXmlRefs FDO_45_ITEMINCOLLECTION

UnitTestUtil.cpp:47:Assertion Test name: 10SchemaTest.testXmlMerge FDO_45_ITEMINCOLLECTION

UnitTestUtil.cpp:47:Assertion Test name: 10SchemaTest.testXmlUnsupported FDO_45_ITEMINCOLLECTION

UnitTestUtil.cpp:47:Assertion Test name: 10SchemaTest.testXmlExternal FDO_45_ITEMINCOLLECTION

UnitTestUtil.cpp:47:Assertion Test name: 10SchemaTest.testXmlSchemaMappings FDO_45_ITEMINCOLLECTION

Failures !!! Run: 102 Failure total: 12 Failures: 12 Errors: 0

[warmerdam]

Applied changesets:

[2511] - fix unit tests to use INT_MAX, INT_MIN for 32bit min/max

[2512] - fix lexer to use INT_MIN/INT_MAX for 32bit values.

[2513] - fix UTF handling to use "unsigned int" for UTF32, not "unsigned long".

Now all fdo core unit tests pass under valgrind, though I am suffering an unexplained crash due to heap corruption when not running under valgrind.

[warmerdam]

Folks,

I mentioned a couple days ago that I was still seeing crashes under some circumstances after my 64bit fixes. I have spent more hours than I wish to contemplate digging into these crashes and have not come up with a particular satisfying answer.

The crashes seemed to be related to corruption of the freelist datastructures maintained by the mt_alloc allocator for stl vector objects (ie. the XalanDOMString).

On Linux with glibc it seems you can define the environment variable GLIBCXX_FORCE_NEW to force a new and delete to be done each time stl container allocate or free memory, instead of using the mt_alloc pool allocator. This seems to avoid the problem. I *thought* when running in this mode I would finally be able to find the bug in the FDO code using valgrind, but things run perfectly clean!

Based on a variety of googling, I am now suspecting that my Ubuntu gcc 3.4 build is suffering from the problem described at:

​http://bugs.donarmstrong.com/cgi-bin/bugreport.cgi?bug=293466

So, if people are running into strange crashes, and tracebacks in gcc show it being deep in the glibc mt_allocator while destroying XalanDOMString's, I would suggest defining the GLIBCXX_FORCE_NEW environment variable. I results in a 20-30% speed reduction in the FDO core unit tests, so there is a performance impact. But things run clean.

I *imagine* this is fixed in newer versions of Debian/Ubuntu. I'm running one generation behind (Ubuntu Breezy Badger).

---

Based on this, I'm now confident with the fdocore 64bit porting, and am closing this report. Still leaving backporting to the descretion of the "powers that be" (ie. Greg.)

[warmerdam]

Changes (except for unit test) incorporated into 3.2.x branch of fdocore as r812.