FileGDB Domains

[aestrada]

I create a FGDB using the following command line args.

ogr2ogr -skipfailures -progress -s_srs EPSG:4326 -t_srs EPSG:4326 -f "FileGDB" data.gdb PG:"$POSTGRESQL" -sql "SELECT id, service, date_taken, date_inserted, url, description, geom FROM export" -nln "January2012_all" -nlt POINT

There are about 10.8 million points that *display* fine in ArcGIS 10.0 but when we go to export them say, by country, the resulting FGDB is empty. It looks like the domain extents are completely out of bounds which is causing this to happen. I glanced at the code and there is nothing overly obvious about why this is happening through GDAL and I also tested the same thing in ArcGIS 10 going from FGDB to SHP. That worked fine. Any thoughts on this one?

Adam

[jpalmer]

Could this be the problem?

 http://forums.arcgis.com/threads/48027-BUG-Table-extents-truncated-breaking-spatial-indexes

If it is our department has just been talking to ESRI and apparently it's only an issue on the linux 64bit version of the API.

Another possibility is we have identified an issue with the default resolution and tolerance values that the ogr FGDB driver sets. These values are only appropriate for metric coordinate systems. If you are using geographic one I would recommend using values like:

XORIGIN: -400 YORIGIN: -400 XYSCALE: 11258999068426.238 XYTOLERANCE: 0.000000008983153

When using a high precision FGDB (which ogr does) this gives the feature class a bounds from -400 to 400 (1/11258999068426.238*2^{53bits=800) in both directions. This allows for coordinates from -180 to 180 and any polygon that may wrap a further 180 over the anti-meridian line.}

At some point the FGDB driver should be patched to do this.

But in the mean time you are try using the layer creation options. e.g:

ogr2ogr .... -lco XORIGIN=400 -lco YORIGIN=-400 -lco XYSCALE=11258999068426.238 -lco XYTOLERANCE=0.000000008983153

Cheers, Jeremy

[aestrada]

Thanks Jeremy! I will test this out ASAP and get back to you with the results!

[rouault]

I tried setting the XYSCALE and other parameters, but it didn't help. I've come with the following workaround. It sets extents that include all geometry bounding boxes, but still rounded to integer values. I cannot verify if this works correctly in ESRI software, especially when spatial indexes are defined. Can anyone confirm ?

r23779 /trunk/gdal/ogr/ogrsf_frmts/filegdb/ (FGdbLayer.cpp ogr_fgdb.h): FileGDB: temptative workaround for problem with Linux 64bit FileGDB SDK that sets invalid extents (#4455)

[aestrada]

On first pass...the workaround using a series of -lco flags did not work. In fact, ArcGIS does not display the data. We would expect to see these

[[[ XY Tolerance: 0.000000008983153 XY Resolution: 0.000000001 XY Domain:

MinX: -400 MinY: -400 MaxX: 9006799.25474099 MaxY: 9006799.25474099

]]]

from a WGS84 FGDB. I am building GDAL trunk now and will test the fix in a few minutes. I suspect that I don't have to set all those flags with the fix in place?

[aestrada]

All,

I rebuilt GDAL from trunk and ran the export from PostGIS to FGDB again and the domain values are still way off and the resulting FGDB is empty.

Adam

[rouault]

I guess I should just revert my r23779 hack if it is of no use ?

[rcoup]

@aestrada

As discussed in IRC, can you please try the following with GDAL trunk >=r23779:

ogr2ogr -skipfailures -progress -s_srs EPSG:4326 -t_srs EPSG:4326 -f "FileGDB" data.gdb \
  PG:"$POSTGRESQL" -sql "SELECT id, service, date_taken, date_inserted, url, description, geom FROM export" -nln "January2012_all" \
  -nlt POINT -lco XORIGIN=400 -lco YORIGIN=-400 -lco XYSCALE=11258999068426.238 \
  -lco XYTOLERANCE=0.000000008983153

Then... try your country export in ArcMap? again and see if it ends up empty. If you can grab the following bits of data (from the FGDB produced by GDAL) that would be great:

• The extent from ogrinfo data.gdb -so -al
• In ArcMap?, the layer extent (layer Properties... Source)
• In ArcMap?, the Feature Dataset XY Tolerance, XY Resolution, and XY Domain values (feature dataset properties in the Catalog pane)
• In ArcMap?, the Feature Class Tolerance, Resolution, and Domain values (feature class properties in the Catalog pane)

Then try deleting the spatial index from the feature class (feature class properties in the Catalog pane... Indexes... Delete) and re-doing your country export in ArcMap? and seeing whether it still ends up empty.

Thanks!

[rcoup]

After some more digging with @aestrada:

1. Export some data from GDAL as FGDB (trunk >=r23779)
2. Open in ArcMap?
3. Select some features
4. Export into a new feature-class in the *same* FileGDB (produced by GDAL)
5. No features :(

(ArcMap? 10.0, Windows Vista 32bit, GDAL is on Linux/64)

Seems to work fine exporting into a new FileGDB, just not the same one.

@jpalmer - any chance you can test/reproduce?

[jpalmer]

When I first tried r23779 over the weekend I got seg faults and corrupted extents. Will do some more testing today and get back to you all.

[rouault]

jpalmer: just in case : make sure you 'make clean' before rebuilding. If you still observe seg faults after, hopefully you can find a way for me to reproduce them, or I'll just revert that horrible workaround that I dislike more and more each time I see it. I'm afraid it just hides a bit more serious issues in the SDK that only ESRI can fix.

[jpalmer]

rouault: Ok thanks - will do. Just to let you know we have been in contact with ESRI and a fix is coming soon. So hopefully this patch is not required. However in saying that we are still keen to get an immediate workaround for our purposes.

[aestrada]

First test...

ogrinfo -so -al

INFO: Open of `test.gdb'
      using driver `FileGDB' successful.

Layer name: January2012_all
Geometry: Point
Feature Count: 10000
Extent: (-80.000000, -35.000000) - (1638521.000000, 61.000000)
Layer SRS WKT:
GEOGCS["WGS 84",
    DATUM["WGS_1984",
        SPHEROID["WGS 84",6378137,298.257223563,
            AUTHORITY["EPSG","7030"]],
        AUTHORITY["EPSG","6326"]],
    PRIMEM["Greenwich",0,
        AUTHORITY["EPSG","8901"]],
    UNIT["degree",0.0174532925199433,
        AUTHORITY["EPSG","9122"]],
    AUTHORITY["EPSG","4326"]]
FID Column = OBJECTID
Geometry Column = SHAPE
id: Integer (0.0)
service: String (0.0)
date_taken: DateTime (0.0)
date_inserted: DateTime (0.0)
url: String (0.0)
description: String (0.0)

Replying to rcoup:

@aestrada As discussed in IRC, can you please try the following with GDAL trunk >=r23779: {{{ ogr2ogr -skipfailures -progress -s_srs EPSG:4326 -t_srs EPSG:4326 -f "FileGDB" data.gdb \ PG:"$POSTGRESQL" -sql "SELECT id, service, date_taken, date_inserted, url, description, geom FROM export" -nln "January2012_all" \ -nlt POINT -lco XORIGIN=400 -lco YORIGIN=-400 -lco XYSCALE=11258999068426.238 \ -lco XYTOLERANCE=0.000000008983153 }}} Then... try your country export in ArcMap? again and see if it ends up empty. If you can grab the following bits of data (from the FGDB produced by GDAL) that would be great: * The extent from ogrinfo data.gdb -so -al * In ArcMap?, the layer extent (layer Properties... Source) * In ArcMap?, the Feature Dataset XY Tolerance, XY Resolution, and XY Domain values (feature dataset properties in the Catalog pane) * In ArcMap?, the Feature Class Tolerance, Resolution, and Domain values (feature class properties in the Catalog pane) Then try deleting the spatial index from the feature class (feature class properties in the Catalog pane... Indexes... Delete) and re-doing your country export in ArcMap? and seeing whether it still ends up empty. Thanks!

[aestrada]

Here is the FGDB exported directly from OGR.  http://goo.gl/Ea6ZS

[rcoup]

k, an update:

• the patch from r23779 definitely resolves the extent issue.
• unfortunately, despite ESRI's assurances, extents are also broken on FileGDBs produced by the API under 32bit Linux. The attached patch (attachment:4455.32bitAsWell.patch , against r23855) removes the 64-bit check.
• if you mess with XORIGIN, YORIGIN, XYSCALE, and XYTOLERANCE then you can get broken domains in the FileGDB, which breaks all sorts of other stuff in ArcMap?, often in subtle ways (like selecting & exporting features). I've added a doc patch (attachment:4455.doc.patch ) which might help to clear it up. Really OGR should probably try to select sensible defaults, but ideally we need to figure out where ESRI gets their XORIGIN values from per-coordinate system

[jpalmer]

This patch implement sensible defaults for XORIGIN, YORIGIN, XYSCALE, and XYTOLERANCE parameters for geographic coordinate systems

[rouault]

r23860 /trunk/gdal/ogr/ogrsf_frmts/filegdb/ogr_fgdb.h: FileGDB: use EXTENT_WORKAROUND in Linux 32 bit case too (#4455)

r23861 /trunk/gdal/ogr/ogrsf_frmts/filegdb/drv_filegdb.html: FileGDB: enhance doc to add precisions about XYTOLERANCE, ZTOLERANCE, XORIGIN, YORIGIN, ZORIGIN, XYSCALE, ZSCALE parameters (#4455)

r23862 /trunk/gdal/ogr/ogrsf_frmts/filegdb/FGdbLayer.cpp: FileGDB : use more sensible default values for XYTOLERANCE, ZTOLERANCE, XORIGIN, YORIGIN, ZORIGIN, XYSCALE, ZSCALE parameters for geographic SRS (#4455)

--> I just modified your patch to test for poSRS != NULL. Not sure if it makes sense to produce FGDBs without SRS, but if you tried, it was causing a segfault (triggered by a step of the autotest suite. Actually not really visible at first, but more now with r23857).

[rcoup]

attachment:domain_cs2.diff is an improvement on jpalmer's patch:

• uses default ArcGIS values for scale/tolerance. Importantly, XYSCALE defaults to 10000 rather than 1000000000 (which will break GDBs)
• uses the same algorithm described in the ArcMap? help to use the default tolerance of 1mm (or equivalent) and a default scale of 10x that.
• doc improvements to describe that they're OGRs defaults too

[rouault]

domain_cs2.diff applied as :

r23864 /trunk/gdal/ogr/ogrsf_frmts/filegdb/ (FGdbLayer.cpp drv_filegdb.html): FileGDB: use more appropriate default values for tolerance and scale parameters (fix r23862, #4455)

[rouault]

r23888 /trunk/gdal/ogr/ogrsf_frmts/filegdb/FGdbLayer.cpp: Fix issue with s_xyscale that got garbage on Windows due to bad scope of the variable (#4455)

[rcoup]

r23925 /trunk/gdal/ogr/ogrsf_frmts/filegdb/FGdbLayer.cpp: FileGDB: get default tolerances and scales around the right way. Use correct calculated default for ZSCALE. (#4455)

[rcoup]

This has been working well for us in production with 1.1.

It's now been fixed upstream with the  FileGDB API 1.2 release. I've confirmed it resolves the problem too.

Marking as fixed, it'd be nice to #ifdefine it, but they need to add some API version #defines/calls.