FGdbLayer doesn't use SetWriteLock() and FreeWriteLock()

[borisdogadov]

This code is present in FGdbLayer::CommitTransaction() and FGdbLayer::StartTransaction(), but both those functions are commented out.

There is a bug when writing more then 1000 features (with point geometry) sometimes OGR.Layer.CreateFeature throws an exception "Workspace or data source is read only.(-2147220893)"... This behavior is easily reproduced in ESRI FileGDB_API_VS2008_1_1, and is solved by calling SetWriteLock() and FreeWriteLock().

So any reason this code it commented out?

[Even Rouault]

Paul,

svn blame would tend to show that you added (in commented form) the StartTransaction() / CommitTransaction() in r22944 , so perhaps you have some experience with that.

[pramsey]

My problem was that I started adding in the code, then read the documentation for the GDAL Commit/Start methods. The doco makes it clear that the semantics are actually RDBMS semantics, while all I could offer in the method were some FGDB sugar for bulk-loading, not anything capable of providing a real rollback. So it seemed misleading to add those FGDB calls in those places. Is there a better place in GDAL for a "start bulk load, end bulk load" cycle to be exposed?

[Even Rouault]

Paul, no, I don't see a standard way in the OGR API to ask for bulk loading. The PG driver has a configuration option PG_USE_COPY for using COPY instead of INSERT. Perhaps a FGDB_BULK_LOAD configuration option could be used to ask for SetWriteLock()/FreeWriteLock().

But the issue here is that it seems that using SetWriteLock()/FreeWriteLock() seems to be necessary sometimes. Although I don't see any problem inserting hundreds of thousands of features on Linux without using those lock functions. Perhaps an issue with Windows ?

borisdogadov : are you writing on a local disk (to be opposed to network storage) ? You could perhaps try reporting your issue to the ESRI forum where people could have better clues.

[borisdogadov]

Hi, thanks for your reply,

I'm writing on a local disk, and the problem mainly occurs with features with Point geometry. I've posted a question in ESRI forum, and they recommend to use those functions, also for performance improvements

Here: ​http://forums.arcgis.com/threads/46848-Table.Insert-fails-with-quot-Workspace-or-data-source-is-read-only.%28-2147220893%29-quot

And here: ​http://forums.arcgis.com/threads/36881-the-performance-FGDB-API-amp-Arcobjects

So maybe FGDB_BULK_LOAD is a good idea?

[Even Rouault]

borisdogadov,

I've just tested the following script which creates 10,000 points with FileGDB 1.1 (Visual Studio 2008 / VC9) and it works without any problem. Does it work for you ? If not, strange. If yes, are you able to modify it to the point where it triggers the "Workspace or data source is read only.(-2147220893)" error. Are you also using latest trunk version (or 1.9.0RC2 equivalently) ?

from osgeo import ogr
from osgeo import osr
import shutil
try:
	shutil.rmtree('test.gdb')
except:
	pass

drv = ogr.GetDriverByName('FileGDB').CreateDataSource('test.gdb')
sr = osr.SpatialReference()
sr.ImportFromEPSG(4326)
lyr = drv.CreateLayer('test', geom_type = ogr.wkbPoint, srs = sr)
lyr.CreateField(ogr.FieldDefn('id', ogr.OFTInteger))
for i in range(10000):
	feat = ogr.Feature(lyr.GetLayerDefn())
	feat.SetField(0, i)
	geom = ogr.CreateGeometryFromWkt('POINT(0 1)')
	feat.SetGeometry(geom)
	lyr.CreateFeature(feat)
	feat = None
ds = None

[borisdogadov]

Hi, The gdal version that I'm using is "GDAL 1.9.0 beta2" and FileGDB version 1.1.0.129 I've wrote your script in c# and it reproduces this bug immediately:

      OSGeo.OGR.Driver driver = OSGeo.OGR.Ogr.GetDriverByName("FileGDB");
      SpatialReference sr = new SpatialReference(OSGeo.OSR.Osr.SRS_WKT_WGS84);
      
      using (OSGeo.OGR.DataSource dataSource = driver.CreateDataSource(@"c:\temp\testgdb.gdb", null)) 
      using (OSGeo.OGR.Layer layer = dataSource.CreateLayer("TestLayer", sr,
                                                            wkbGeometryType.wkbPoint,
                                                            null))
      {
        layer.CreateField(new FieldDefn("id", FieldType.OFTInteger), 1);
        layer.StartTransaction();
        OSGeo.OGR.Feature feature;
               
        for (int i = 0; i < 10000; i++)
        {
          feature = new OSGeo.OGR.Feature(layer.GetLayerDefn());  
          feature.SetField(0, i);
          
          string wkt = "POINT(0 1)";
          Geometry g = Ogr.CreateGeometryFromWkt(ref wkt, sr);
          feature.SetGeometry(g);
          
          try
          {
            layer.CreateFeature(feature);
          }
          catch (Exception e)
          {
            string s = e.Message;
          }
        }      
        layer.CommitTransaction();
      }

So I get the following exception: "System.ApplicationException: Error: Failed at writing Row to Table in CreateFeature. (Workspace or data source is read only.) at OSGeo.OGR.Layer.CreateFeature(Feature feature)"

Also this behavior is consistent on my machine when using filegdbapi.dll directly and writing this test:

    int numPoints = 5000;
    wcout << "Writing " << numPoints << " points to the database." << endl;
    fgdbError hr;
    wstring errorText;
    Geodatabase geodatabase;
    if ((hr = OpenGeodatabase(L"SomeFile.gdb", geodatabase)) != S_OK)
    {
      wcout << "An error occurred while opening the geodatabase." << endl;
      ErrorInfo::GetErrorDescription(hr, errorText);
      wcout << errorText << "(" << hr << ")." << endl;
      return -1;
    }
    
    Table table;
    if ((hr = geodatabase.OpenTable(L"\\Points", table)) != S_OK)
    {
      wcout << "An error occurred while opening the geodatabase." << endl;
      ErrorInfo::GetErrorDescription(hr, errorText);
      wcout << errorText << "(" << hr << ")." << endl;
      return -1;
    }
    
    Row row;
    PointShapeBuffer psb;
    Point* point;
    
    tm time;
    
    for (int i = 0; i < numPoints; i++)
    {
      table.CreateRowObject(row);
      
      row.SetString(L"TextField", L"Random text string.");
      row.SetDouble(L"DoubleField1", 1);
      row.SetDouble(L"DoubleField2", 2);
      row.SetDate(L"DateField",  time);
      row.SetInteger(L"IntegerField1", 3);
      row.SetInteger(L"IntegerField2", 4);

      double x = rand() % 360;
      double y = rand() % 180;
    
      psb.Setup(ShapeType::shapePoint);
      psb.GetPoint(point);
      point->x = x;
      point->y = y;
      
      row.SetGeometry(psb);      
      hr = table.Insert(row);
      
      if (hr != S_OK)
      {
        ErrorInfo::GetErrorDescription(hr, errorText);
        wcout << L"Point #" << i << " " << errorText << "(" << hr << ")." << endl;
      }      

      //wcout << "end " << i << endl;
    }    
    
    geodatabase.CloseTable(table);
    CloseGeodatabase(geodatabase);
    wcout << "OK" << endl;

I've tried running this test on three different machines and reproduced the exception on two of them. I guess it is some ESRI FGDB api bug, and the way to bypass it is to use the Set/Free methods (also it should speed up the writing).

Thanks for you help.

[Even Rouault]

r23701 /trunk/ (4 files in 2 dirs): FileGDB: add the FGDB_BULK_LOAD configuration option that can be set to YES to speed-up bulk feature loading (or to workaround a likely fgdb sdk bug) (#4420)

[borisdogadov]

Hi!

Thanks for you help!

I've tried to check your solution with the latest build (1/8/2012 4:48 PM) but there is a crash while trying open any GDB.

Ogrinfo on any GDB crushes also, does it happen to anybody else? I don't have pdb, but VS points that exception is at ogr_FileGDB.dll.

[Even Rouault]

Did you "make /f makefile.vc clean" before rebuilding ? The r23701 commit changes the size of the FGdbLayer, so that could explain weird behaviour.

[borisdogadov]

Ok, thanks! I can confirm that the fix does resolve the issue.

[Even Rouault]

Backported in branches/1.9 (r25013)