.NET: Exceptions being thrown on IInserts/IUpdates involving large datasets

[MaksimS]

Managed wrapper seems to throw an error while performing perpetual IInserts/IUpdates involving large datasets (i.e. tens of thousands of feature instances). This keeps happening since god-knows-when, but let's set it to version 3.2.0 (continued to appear in 3.3.0 also).

Exception being thrown:

- System.AccessViolationExceptions ("Attempted to read or write protected memory. This is often an indication that other memory is corrupt") thrown at
OSGeo.FDO.Runtime.Disposable.Finalize() (Target Site: void
ReleaseUnmanagedObject() )
- C++ pure virtual function call errors.
- Memory allocation failed errors.

There are also other types of exceptions fired here and there, scattered among other related projects (i.e. MapGuide) - possibly related to this: ​http://trac.osgeo.org/mapguide/ticket/323

It's general opinion (me, Clayton) that something's buggy with managed wrapper and garbage collector - it's not disposing its underlying unmanaged resources in the correct order. As Clayton correctly pointed out: "...and that this problem will occur when the garbage collector comes around - hence the pseudo-randomness...", therefor a fuzziness on properly pinpointing source of the problem.

[gregboone]

Test Details:

Unit Test:

        public void Test_LargeDataVolumeInsert()
        {
            IConnection mConnection = base.ConnectionInitialized;

            mConnection.ConnectionString = @"File=..\..\..\..\TestData\PARCEL_NEW.SDF";
            ConnectionState state = mConnection.Open();

            IInsert insertCmd = (IInsert)mConnection.CreateCommand(CommandType.CommandType_Insert);

            insertCmd.SetFeatureClassName("MyClass");
            PropertyValueCollection propVals = insertCmd.PropertyValues;

            for (Int32 counter = 0; counter < System.Int32.MaxValue - 1; counter++)
            {
                propVals.Clear();
                Expression expr = Expression.Parse("'ABCDEFGHIJKLOMOP'");
                PropertyValue propVal = new PropertyValue("NotNullString", (ValueExpression)expr);
                propVals.Add(propVal);
                expr = Expression.Parse("'Poor'");
                propVal = new PropertyValue("StringRange", (ValueExpression)expr);
                propVals.Add(propVal);
                expr = ParseByDataType("10000", DataType.DataType_Int32);
                propVal = new PropertyValue("Int32Range", (ValueExpression)expr);
                propVals.Add(propVal);
                expr = Expression.Parse("'Down'");
                propVal = new PropertyValue("StringEnum", (ValueExpression)expr);
                propVals.Add(propVal);
                expr = Expression.Parse("GEOMFROMTEXT('LINESTRING XY (100000.0 100000.0, 200000.0 200000.0, 100000.0 300000.0)')");
                propVal = new PropertyValue("Geom", (ValueExpression)expr);
                propVals.Add(propVal);
                IFeatureReader reader = insertCmd.Execute();
                reader.Close();
            }

            Console.WriteLine("Test_LargeDataVolumeInsert runs successfully !");
        }

Call Stack:

	SDFProvider.dll!sqlite3pager_ref(0xfeeefeee)  Line 2174	C
 	SDFProvider.dll!initPage(0x068459f0, 0x06839128)  Line 1304	C
 	SDFProvider.dll!getAndInitPage(0x04b5aee8, 36, 0x0012bd00, 0x06839128)  Line 1417	C
 	SDFProvider.dll!moveToChild(0x06800c08, 36)  Line 3090	C
 	SDFProvider.dll!sqlite3BtreeMoveto(0x06800c08, 0x00000000, 321, 0x0012bd80)  Line 3394	C
 	SDFProvider.dll!SQLiteCursor::move_to(4, 0x0012be08, false)  Line 63	C++
 	SDFProvider.dll!SQLiteTable::get(0x00000000, 0x0012bde0, 0x0012bde8, 0, true)  Line 622	C++
 	SDFProvider.dll!SdfRTree::RetrieveNode({...}, 321, true)  Line 133	C++
 	SDFProvider.dll!SdfRTree::InsertRect2({...}, {...}, {...}, 0, {...}, 0)  Line 261	C++
 	SDFProvider.dll!SdfRTree::InsertRect2({...}, {...}, {...}, 0, {...}, 0)  Line 261	C++
 	SDFProvider.dll!SdfRTree::Insert({...}, 0, {...}, 0)  Line 200	C++
>	SDFProvider.dll!SdfInsert::Execute()  Line 272	C++
 	[Managed to Native Transition]	
 	OSGeo.FDO.dll!OSGeo::FDO::Commands::Feature::IInsertImp::Execute() Line 77	C++
 	mgUnitTest.exe!unit_test.ProviderTests.SchemaTests.Test_LargeDataVolumeInsert() Line 207	C#


Local Function:

/*
** Increment the reference count for a page.  The input pointer is
** a reference to the page data.
*/
int sqlite3pager_ref(void *pData){
  PgHdr *pPg = DATA_TO_PGHDR(pData);
  page_ref(pPg);
  return SQLITE_OK;
}

Autos:

  pData	0xfeeefeee	void *
+ pPg	0xfeeefeb6 {pPager=??? pgno=??? pNextHash=??? ...}	PgHdr *

[gregboone]

Looks like a managed garbage collection/SDF Feature reader interaction issue.

If I manually dispose the reader by adding

reader.Dispose();

to the above for loop, then all is fine. I assume the garbage collector is not disposing the reader objects at a fast enough rate to keep up with the new ones being created. I am investigating if there a way to free all the SDF resources attached to the feature reader on a call to Close().

[MaksimS]

This is definatelly affecting related projects, I just noticed following thread on MapGuide wiki, seems so related to the quirks I've been experiencing on MGE:

​http://www.nabble.com/MGOS-2.0%3A-MgUpdateFeatures-broken-to16467463s16610.html#a16491260

[badreddinekaroui]

Moved the cleanup code from the destructor to Close.