Opened 11 years ago

Closed 11 years ago

Last modified 10 years ago

#4420 closed defect (fixed)

FGdbLayer doesn't use SetWriteLock() and FreeWriteLock()

Reported by: borisdogadov Owned by: warmerdam
Priority: normal Milestone: 1.9.2
Component: OGR_SF Version: unspecified
Severity: normal Keywords: FileGDB
Cc: pramsey, Kyle Shannon

Description

This code is present in FGdbLayer::CommitTransaction() and FGdbLayer::StartTransaction(), but both those functions are commented out.

There is a bug when writing more then 1000 features (with point geometry) sometimes OGR.Layer.CreateFeature throws an exception "Workspace or data source is read only.(-2147220893)"... This behavior is easily reproduced in ESRI FileGDB_API_VS2008_1_1, and is solved by calling SetWriteLock() and FreeWriteLock().

So any reason this code it commented out?

Change History (13)

comment:1 by Even Rouault, 11 years ago

Cc: pramsey added

Paul,

svn blame would tend to show that you added (in commented form) the StartTransaction() / CommitTransaction() in r22944 , so perhaps you have some experience with that.

comment:2 by pramsey, 11 years ago

My problem was that I started adding in the code, then read the documentation for the GDAL Commit/Start methods. The doco makes it clear that the semantics are actually RDBMS semantics, while all I could offer in the method were some FGDB sugar for bulk-loading, not anything capable of providing a real rollback. So it seemed misleading to add those FGDB calls in those places. Is there a better place in GDAL for a "start bulk load, end bulk load" cycle to be exposed?

comment:3 by Even Rouault, 11 years ago

Paul, no, I don't see a standard way in the OGR API to ask for bulk loading. The PG driver has a configuration option PG_USE_COPY for using COPY instead of INSERT. Perhaps a FGDB_BULK_LOAD configuration option could be used to ask for SetWriteLock()/FreeWriteLock().

But the issue here is that it seems that using SetWriteLock()/FreeWriteLock() seems to be necessary sometimes. Although I don't see any problem inserting hundreds of thousands of features on Linux without using those lock functions. Perhaps an issue with Windows ?

borisdogadov : are you writing on a local disk (to be opposed to network storage) ? You could perhaps try reporting your issue to the ESRI forum where people could have better clues.

comment:4 by borisdogadov, 11 years ago

Hi, thanks for your reply,

I'm writing on a local disk, and the problem mainly occurs with features with Point geometry. I've posted a question in ESRI forum, and they recommend to use those functions, also for performance improvements

Here: http://forums.arcgis.com/threads/46848-Table.Insert-fails-with-quot-Workspace-or-data-source-is-read-only.%28-2147220893%29-quot

And here: http://forums.arcgis.com/threads/36881-the-performance-FGDB-API-amp-Arcobjects

So maybe FGDB_BULK_LOAD is a good idea?

comment:5 by Even Rouault, 11 years ago

borisdogadov,

I've just tested the following script which creates 10,000 points with FileGDB 1.1 (Visual Studio 2008 / VC9) and it works without any problem. Does it work for you ? If not, strange. If yes, are you able to modify it to the point where it triggers the "Workspace or data source is read only.(-2147220893)" error. Are you also using latest trunk version (or 1.9.0RC2 equivalently) ?

from osgeo import ogr
from osgeo import osr
import shutil
try:
	shutil.rmtree('test.gdb')
except:
	pass

drv = ogr.GetDriverByName('FileGDB').CreateDataSource('test.gdb')
sr = osr.SpatialReference()
sr.ImportFromEPSG(4326)
lyr = drv.CreateLayer('test', geom_type = ogr.wkbPoint, srs = sr)
lyr.CreateField(ogr.FieldDefn('id', ogr.OFTInteger))
for i in range(10000):
	feat = ogr.Feature(lyr.GetLayerDefn())
	feat.SetField(0, i)
	geom = ogr.CreateGeometryFromWkt('POINT(0 1)')
	feat.SetGeometry(geom)
	lyr.CreateFeature(feat)
	feat = None
ds = None

comment:6 by borisdogadov, 11 years ago

Hi, The gdal version that I'm using is "GDAL 1.9.0 beta2" and FileGDB version 1.1.0.129 I've wrote your script in c# and it reproduces this bug immediately:

      OSGeo.OGR.Driver driver = OSGeo.OGR.Ogr.GetDriverByName("FileGDB");
      SpatialReference sr = new SpatialReference(OSGeo.OSR.Osr.SRS_WKT_WGS84);
      
      using (OSGeo.OGR.DataSource dataSource = driver.CreateDataSource(@"c:\temp\testgdb.gdb", null)) 
      using (OSGeo.OGR.Layer layer = dataSource.CreateLayer("TestLayer", sr,
                                                            wkbGeometryType.wkbPoint,
                                                            null))
      {
        layer.CreateField(new FieldDefn("id", FieldType.OFTInteger), 1);
        layer.StartTransaction();
        OSGeo.OGR.Feature feature;
               
        for (int i = 0; i < 10000; i++)
        {
          feature = new OSGeo.OGR.Feature(layer.GetLayerDefn());  
          feature.SetField(0, i);
          
          string wkt = "POINT(0 1)";
          Geometry g = Ogr.CreateGeometryFromWkt(ref wkt, sr);
          feature.SetGeometry(g);
          
          try
          {
            layer.CreateFeature(feature);
          }
          catch (Exception e)
          {
            string s = e.Message;
          }
        }      
        layer.CommitTransaction();
      }

So I get the following exception: "System.ApplicationException: Error: Failed at writing Row to Table in CreateFeature. (Workspace or data source is read only.) at OSGeo.OGR.Layer.CreateFeature(Feature feature)"

Also this behavior is consistent on my machine when using filegdbapi.dll directly and writing this test:

    int numPoints = 5000;
    wcout << "Writing " << numPoints << " points to the database." << endl;
    fgdbError hr;
    wstring errorText;
    Geodatabase geodatabase;
    if ((hr = OpenGeodatabase(L"SomeFile.gdb", geodatabase)) != S_OK)
    {
      wcout << "An error occurred while opening the geodatabase." << endl;
      ErrorInfo::GetErrorDescription(hr, errorText);
      wcout << errorText << "(" << hr << ")." << endl;
      return -1;
    }
    
    Table table;
    if ((hr = geodatabase.OpenTable(L"\\Points", table)) != S_OK)
    {
      wcout << "An error occurred while opening the geodatabase." << endl;
      ErrorInfo::GetErrorDescription(hr, errorText);
      wcout << errorText << "(" << hr << ")." << endl;
      return -1;
    }
    
    Row row;
    PointShapeBuffer psb;
    Point* point;
    
    tm time;
    
    for (int i = 0; i < numPoints; i++)
    {
      table.CreateRowObject(row);
      
      row.SetString(L"TextField", L"Random text string.");
      row.SetDouble(L"DoubleField1", 1);
      row.SetDouble(L"DoubleField2", 2);
      row.SetDate(L"DateField",  time);
      row.SetInteger(L"IntegerField1", 3);
      row.SetInteger(L"IntegerField2", 4);

      double x = rand() % 360;
      double y = rand() % 180;
    
      psb.Setup(ShapeType::shapePoint);
      psb.GetPoint(point);
      point->x = x;
      point->y = y;
      
      row.SetGeometry(psb);      
      hr = table.Insert(row);
      
      if (hr != S_OK)
      {
        ErrorInfo::GetErrorDescription(hr, errorText);
        wcout << L"Point #" << i << " " << errorText << "(" << hr << ")." << endl;
      }      

      //wcout << "end " << i << endl;
    }    
    
    geodatabase.CloseTable(table);
    CloseGeodatabase(geodatabase);
    wcout << "OK" << endl;

I've tried running this test on three different machines and reproduced the exception on two of them. I guess it is some ESRI FGDB api bug, and the way to bypass it is to use the Set/Free methods (also it should speed up the writing).

Thanks for you help.

comment:7 by Even Rouault, 11 years ago

Component: defaultOGR_SF
Keywords: FileGDB added

r23701 /trunk/ (4 files in 2 dirs): FileGDB: add the FGDB_BULK_LOAD configuration option that can be set to YES to speed-up bulk feature loading (or to workaround a likely fgdb sdk bug) (#4420)

comment:8 by Kyle Shannon, 11 years ago

Cc: Kyle Shannon added

comment:9 by borisdogadov, 11 years ago

Hi!

Thanks for you help!

I've tried to check your solution with the latest build (1/8/2012 4:48 PM) but there is a crash while trying open any GDB.

Ogrinfo on any GDB crushes also, does it happen to anybody else? I don't have pdb, but VS points that exception is at ogr_FileGDB.dll.

comment:10 by Even Rouault, 11 years ago

Did you "make /f makefile.vc clean" before rebuilding ? The r23701 commit changes the size of the FGdbLayer, so that could explain weird behaviour.

comment:11 by borisdogadov, 11 years ago

Ok, thanks! I can confirm that the fix does resolve the issue.

comment:12 by Even Rouault, 11 years ago

Milestone: 2.0.0
Resolution: fixed
Status: newclosed

comment:13 by Even Rouault, 10 years ago

Milestone: 2.0.01.9.2

Backported in branches/1.9 (r25013)

Note: See TracTickets for help on using tickets.