Opened 11 years ago

Closed 10 years ago

#4455 closed defect (fixed)

FileGDB Domains

Reported by: aestrada Owned by: Robert Coup
Priority: high Milestone: 1.9.2
Component: OGR_SF Version: svn-trunk
Severity: major Keywords: fgdb, ogr2ogr, esri, filegdb
Cc: ryanl

Description

I create a FGDB using the following command line args.

ogr2ogr -skipfailures -progress -s_srs EPSG:4326 -t_srs EPSG:4326 -f "FileGDB" data.gdb PG:"$POSTGRESQL" -sql "SELECT id, service, date_taken, date_inserted, url, description, geom FROM export" -nln "January2012_all" -nlt POINT

There are about 10.8 million points that *display* fine in ArcGIS 10.0 but when we go to export them say, by country, the resulting FGDB is empty. It looks like the domain extents are completely out of bounds which is causing this to happen. I glanced at the code and there is nothing overly obvious about why this is happening through GDAL and I also tested the same thing in ArcGIS 10 going from FGDB to SHP. That worked fine. Any thoughts on this one?

Adam

Attachments (4)

4455.32bitAsWell.patch (573 bytes ) - added by Robert Coup 11 years ago.
4455.doc.patch (4.8 KB ) - added by Robert Coup 11 years ago.
domain_cs.patch (1.3 KB ) - added by jpalmer 11 years ago.
This patch implement sensible defaults for XORIGIN, YORIGIN, XYSCALE, and XYTOLERANCE parameters for geographic coordinate systems
domain_cs2.diff (4.9 KB ) - added by Robert Coup 11 years ago.

Download all attachments as: .zip

Change History (29)

comment:1 by jpalmer, 11 years ago

Could this be the problem?

http://forums.arcgis.com/threads/48027-BUG-Table-extents-truncated-breaking-spatial-indexes

If it is our department has just been talking to ESRI and apparently it's only an issue on the linux 64bit version of the API.

Another possibility is we have identified an issue with the default resolution and tolerance values that the ogr FGDB driver sets. These values are only appropriate for metric coordinate systems. If you are using geographic one I would recommend using values like:

XORIGIN: -400 YORIGIN: -400 XYSCALE: 11258999068426.238 XYTOLERANCE: 0.000000008983153

When using a high precision FGDB (which ogr does) this gives the feature class a bounds from -400 to 400 (1/11258999068426.238*253bits=800) in both directions. This allows for coordinates from -180 to 180 and any polygon that may wrap a further 180 over the anti-meridian line.

At some point the FGDB driver should be patched to do this.

But in the mean time you are try using the layer creation options. e.g:

ogr2ogr .... -lco XORIGIN=400 -lco YORIGIN=-400 -lco XYSCALE=11258999068426.238 -lco XYTOLERANCE=0.000000008983153

Cheers, Jeremy

comment:2 by aestrada, 11 years ago

Thanks Jeremy! I will test this out ASAP and get back to you with the results!

comment:3 by Even Rouault, 11 years ago

I tried setting the XYSCALE and other parameters, but it didn't help. I've come with the following workaround. It sets extents that include all geometry bounding boxes, but still rounded to integer values. I cannot verify if this works correctly in ESRI software, especially when spatial indexes are defined. Can anyone confirm ?

r23779 /trunk/gdal/ogr/ogrsf_frmts/filegdb/ (FGdbLayer.cpp ogr_fgdb.h): FileGDB: temptative workaround for problem with Linux 64bit FileGDB SDK that sets invalid extents (#4455)

comment:4 by aestrada, 11 years ago

On first pass...the workaround using a series of -lco flags did not work. In fact, ArcGIS does not display the data. We would expect to see these

[[[ XY Tolerance: 0.000000008983153 XY Resolution: 0.000000001 XY Domain:

MinX: -400 MinY: -400 MaxX: 9006799.25474099 MaxY: 9006799.25474099

]]]

from a WGS84 FGDB. I am building GDAL trunk now and will test the fix in a few minutes. I suspect that I don't have to set all those flags with the fix in place?

comment:5 by aestrada, 11 years ago

All,

I rebuilt GDAL from trunk and ran the export from PostGIS to FGDB again and the domain values are still way off and the resulting FGDB is empty.

Adam

comment:6 by Even Rouault, 11 years ago

I guess I should just revert my r23779 hack if it is of no use ?

comment:7 by Robert Coup, 11 years ago

@aestrada

As discussed in IRC, can you please try the following with GDAL trunk >=r23779:

ogr2ogr -skipfailures -progress -s_srs EPSG:4326 -t_srs EPSG:4326 -f "FileGDB" data.gdb \
  PG:"$POSTGRESQL" -sql "SELECT id, service, date_taken, date_inserted, url, description, geom FROM export" -nln "January2012_all" \
  -nlt POINT -lco XORIGIN=400 -lco YORIGIN=-400 -lco XYSCALE=11258999068426.238 \
  -lco XYTOLERANCE=0.000000008983153

Then... try your country export in ArcMap again and see if it ends up empty. If you can grab the following bits of data (from the FGDB produced by GDAL) that would be great:

  • The extent from ogrinfo data.gdb -so -al
  • In ArcMap, the layer extent (layer Properties... Source)
  • In ArcMap, the Feature Dataset XY Tolerance, XY Resolution, and XY Domain values (feature dataset properties in the Catalog pane)
  • In ArcMap, the Feature Class Tolerance, Resolution, and Domain values (feature class properties in the Catalog pane)

Then try deleting the spatial index from the feature class (feature class properties in the Catalog pane... Indexes... Delete) and re-doing your country export in ArcMap and seeing whether it still ends up empty.

Thanks!

comment:8 by Robert Coup, 11 years ago

Cc: ryanl added
Component: defaultOGR_SF
Keywords: filegdb added
Version: unspecifiedsvn-trunk

comment:9 by Robert Coup, 11 years ago

After some more digging with @aestrada:

  1. Export some data from GDAL as FGDB (trunk >=r23779)
  2. Open in ArcMap
  3. Select some features
  4. Export into a new feature-class in the *same* FileGDB (produced by GDAL)
  5. No features :(

(ArcMap 10.0, Windows Vista 32bit, GDAL is on Linux/64)

Seems to work fine exporting into a new FileGDB, just not the same one.

@jpalmer - any chance you can test/reproduce?

comment:10 by jpalmer, 11 years ago

When I first tried r23779 over the weekend I got seg faults and corrupted extents. Will do some more testing today and get back to you all.

comment:11 by Even Rouault, 11 years ago

jpalmer: just in case : make sure you 'make clean' before rebuilding. If you still observe seg faults after, hopefully you can find a way for me to reproduce them, or I'll just revert that horrible workaround that I dislike more and more each time I see it. I'm afraid it just hides a bit more serious issues in the SDK that only ESRI can fix.

comment:12 by jpalmer, 11 years ago

rouault: Ok thanks - will do. Just to let you know we have been in contact with ESRI and a fix is coming soon. So hopefully this patch is not required. However in saying that we are still keen to get an immediate workaround for our purposes.

in reply to:  7 comment:13 by aestrada, 11 years ago

First test...

ogrinfo -so -al

INFO: Open of `test.gdb'
      using driver `FileGDB' successful.

Layer name: January2012_all
Geometry: Point
Feature Count: 10000
Extent: (-80.000000, -35.000000) - (1638521.000000, 61.000000)
Layer SRS WKT:
GEOGCS["WGS 84",
    DATUM["WGS_1984",
        SPHEROID["WGS 84",6378137,298.257223563,
            AUTHORITY["EPSG","7030"]],
        AUTHORITY["EPSG","6326"]],
    PRIMEM["Greenwich",0,
        AUTHORITY["EPSG","8901"]],
    UNIT["degree",0.0174532925199433,
        AUTHORITY["EPSG","9122"]],
    AUTHORITY["EPSG","4326"]]
FID Column = OBJECTID
Geometry Column = SHAPE
id: Integer (0.0)
service: String (0.0)
date_taken: DateTime (0.0)
date_inserted: DateTime (0.0)
url: String (0.0)
description: String (0.0)

Replying to rcoup:

@aestrada

As discussed in IRC, can you please try the following with GDAL trunk >=r23779:

ogr2ogr -skipfailures -progress -s_srs EPSG:4326 -t_srs EPSG:4326 -f "FileGDB" data.gdb \
  PG:"$POSTGRESQL" -sql "SELECT id, service, date_taken, date_inserted, url, description, geom FROM export" -nln "January2012_all" \
  -nlt POINT -lco XORIGIN=400 -lco YORIGIN=-400 -lco XYSCALE=11258999068426.238 \
  -lco XYTOLERANCE=0.000000008983153

Then... try your country export in ArcMap again and see if it ends up empty. If you can grab the following bits of data (from the FGDB produced by GDAL) that would be great:

  • The extent from ogrinfo data.gdb -so -al
  • In ArcMap, the layer extent (layer Properties... Source)
  • In ArcMap, the Feature Dataset XY Tolerance, XY Resolution, and XY Domain values (feature dataset properties in the Catalog pane)
  • In ArcMap, the Feature Class Tolerance, Resolution, and Domain values (feature class properties in the Catalog pane)

Then try deleting the spatial index from the feature class (feature class properties in the Catalog pane... Indexes... Delete) and re-doing your country export in ArcMap and seeing whether it still ends up empty.

Thanks!

comment:14 by aestrada, 11 years ago

Here is the FGDB exported directly from OGR. http://goo.gl/Ea6ZS

comment:15 by Robert Coup, 11 years ago

k, an update:

  • the patch from r23779 definitely resolves the extent issue.
  • unfortunately, despite ESRI's assurances, extents are also broken on FileGDBs produced by the API under 32bit Linux. The attached patch (attachment:4455.32bitAsWell.patch, against r23855) removes the 64-bit check.
  • if you mess with XORIGIN, YORIGIN, XYSCALE, and XYTOLERANCE then you can get broken domains in the FileGDB, which breaks all sorts of other stuff in ArcMap, often in subtle ways (like selecting & exporting features). I've added a doc patch (attachment:4455.doc.patch) which might help to clear it up. Really OGR should probably try to select sensible defaults, but ideally we need to figure out where ESRI gets their XORIGIN values from per-coordinate system

by Robert Coup, 11 years ago

Attachment: 4455.32bitAsWell.patch added

by Robert Coup, 11 years ago

Attachment: 4455.doc.patch added

by jpalmer, 11 years ago

Attachment: domain_cs.patch added

This patch implement sensible defaults for XORIGIN, YORIGIN, XYSCALE, and XYTOLERANCE parameters for geographic coordinate systems

comment:16 by Even Rouault, 11 years ago

r23860 /trunk/gdal/ogr/ogrsf_frmts/filegdb/ogr_fgdb.h: FileGDB: use EXTENT_WORKAROUND in Linux 32 bit case too (#4455)

r23861 /trunk/gdal/ogr/ogrsf_frmts/filegdb/drv_filegdb.html: FileGDB: enhance doc to add precisions about XYTOLERANCE, ZTOLERANCE, XORIGIN, YORIGIN, ZORIGIN, XYSCALE, ZSCALE parameters (#4455)

r23862 /trunk/gdal/ogr/ogrsf_frmts/filegdb/FGdbLayer.cpp: FileGDB : use more sensible default values for XYTOLERANCE, ZTOLERANCE, XORIGIN, YORIGIN, ZORIGIN, XYSCALE, ZSCALE parameters for geographic SRS (#4455)

--> I just modified your patch to test for poSRS != NULL. Not sure if it makes sense to produce FGDBs without SRS, but if you tried, it was causing a segfault (triggered by a step of the autotest suite. Actually not really visible at first, but more now with r23857).

by Robert Coup, 11 years ago

Attachment: domain_cs2.diff added

comment:17 by Robert Coup, 11 years ago

attachment:domain_cs2.diff is an improvement on jpalmer's patch:

  • uses default ArcGIS values for scale/tolerance. Importantly, XYSCALE defaults to 10000 rather than 1000000000 (which will break GDBs)
  • uses the same algorithm described in the ArcMap help to use the default tolerance of 1mm (or equivalent) and a default scale of 10x that.
  • doc improvements to describe that they're OGRs defaults too

comment:18 by Even Rouault, 11 years ago

domain_cs2.diff applied as :

r23864 /trunk/gdal/ogr/ogrsf_frmts/filegdb/ (FGdbLayer.cpp drv_filegdb.html): FileGDB: use more appropriate default values for tolerance and scale parameters (fix r23862, #4455)

comment:19 by Even Rouault, 11 years ago

r23888 /trunk/gdal/ogr/ogrsf_frmts/filegdb/FGdbLayer.cpp: Fix issue with s_xyscale that got garbage on Windows due to bad scope of the variable (#4455)

comment:20 by Robert Coup, 11 years ago

Owner: changed from warmerdam to Robert Coup

r23925 /trunk/gdal/ogr/ogrsf_frmts/filegdb/FGdbLayer.cpp: FileGDB: get default tolerances and scales around the right way. Use correct calculated default for ZSCALE. (#4455)

comment:21 by Robert Coup, 11 years ago

Resolution: fixed
Status: newclosed

This has been working well for us in production with 1.1.

It's now been fixed upstream with the FileGDB API 1.2 release. I've confirmed it resolves the problem too.

Marking as fixed, it'd be nice to #ifdefine it, but they need to add some API version #defines/calls.

comment:22 by aestrada, 10 years ago

Resolution: fixed
Severity: normalmajor
Status: closedreopened

All,

Bad news...This is broken again with the latest rev. of the 64bit GDAL Driver on Windows. The previous error we were seeing was happening on a 64bit Linux box. Has anyone else had domain errors on Windows?

Adam

comment:23 by Robert Coup, 10 years ago

Hi aestrada,

So this is the 1.2 release of the FileGDB API from ESRI? Which Windows version/build-environment/gdal-version?

Can you get the code from the ZIP file at http://forums.arcgis.com/threads/48027-BUG-Table-extents-truncated-breaking-spatial-indexes?p=164383&viewfull=1#post164383 to compile and fail?

comment:24 by aestrada, 10 years ago

@rcoup,

This is the latest version of the ESRI driver which is version 1.2 and its running on 64bit Windows (Server 2008 and Windows 7). GDAL was built using the MapServer build scripts using GDAL 1.9.1 or whatever the latest release there is. Could it be that these patches are in there? Checking now...http://www.gisinternals.com/sdk/PackageInfo.aspx?file=release-1500-x64-gdal-1-9-1-mapserver-6-0-3.zip Looks like the 1.9.1 version of GDAL might not have these fixes in there. BUT, I have not tried to build from trunk using the same driver on Windows. The key is to get it to work in 64bit mode. Thoughts?

Adam

comment:25 by Even Rouault, 10 years ago

Milestone: 1.9.2
Resolution: fixed
Status: reopenedclosed

branches/1.9 r25012 "FileGDB: backport trunk fixes for tolerance and scale parameters when writing a SpatialReference definition (#4455)"

Note: See TracTickets for help on using tickets.