Opened 5 years ago

Last modified 3 years ago

#4378 assigned defect

Find and fix errors in ESRI WKT morphing

Reported by: etourigny Owned by: etourigny
Priority: normal Milestone:
Component: OGR_SRS Version: unspecified
Severity: normal Keywords: esri wkt morph
Cc: warmerdam, Kyle Shannon

Description

ESRI WKT morphing magic needs some improvement and testing with actual data. Many ESRI EKT .prj files have been collected in #4345 .

Added simple autotest to osr/osr_esri.py and support data in r23510.

For each input ESRI WKT, the test morphs from and to ESRI and checks for matching SRS (with IsSame?()) and also for matching WKT (non fatal as almost all fail this test due to precision differences).

Attaching first results, there are quite a few errors, mostly due to different or missing datums.

Attachments (6)

esri_gcs_srs.txt (8.7 KB) - added by etourigny 5 years ago.
esri_pcs_srs.txt (41.9 KB) - added by etourigny 5 years ago.
log file with errors in PCS defifnitions
add_esri_column.py (4.3 KB) - added by etourigny 5 years ago.
esri_datum_override.csv (514 bytes) - added by etourigny 5 years ago.
esri_gcs.csv (147.7 KB) - added by etourigny 5 years ago.
gdal_datum.csv (182.1 KB) - added by etourigny 5 years ago.

Download all attachments as: .zip

Change History (17)

Changed 5 years ago by etourigny

Attachment: esri_gcs_srs.txt added

Changed 5 years ago by etourigny

Attachment: esri_pcs_srs.txt added

log file with errors in PCS defifnitions

comment:1 Changed 5 years ago by etourigny

Status: newassigned

comment:2 Changed 5 years ago by etourigny

Test files in osr/data have been obtained from ESRI, transformed to a CSV file (semi-colon separator) and gzipped for space. The file data/esri_extra.csv is to add any other definitions.

comment:3 Changed 5 years ago by Even Rouault

data/esri_extra.csv has not been commited.

And I get 'fail' tests. Perhaps due to the fact that you have local changes in your ogr/ogr_srs_esri.cpp

  TEST: osr_esri_21 ... WARNING: Failed 738 WKT tests, see file tmp/esri_gcs_wkt.txt
fail
    line 824: ERROR: Failed 26 SRS tests, see file tmp/esri_gcs_srs.txt
  TEST: osr_esri_22 ... WARNING: Failed 4268 WKT tests, see file tmp/esri_pcs_wkt.txt
fail
    line 824: ERROR: Failed 48 SRS tests, see file tmp/esri_pcs_srs.txt
  TEST: osr_esri_23 ... fail
    line 767: input file data/esri_extra.csv does not exist

Test Script: osr_esri
Succeeded: 20
Failed:    3 (0 blew exceptions)
Skipped:   0
Expected fail:0

And the _wtk.txt files are always empty :

-rw-r--r-- 1 even even  8926 2011-12-10 18:18 esri_gcs_srs.txt
-rw-r--r-- 1 even even     0 2011-12-10 18:18 esri_gcs_wkt.txt
-rw-r--r-- 1 even even 42953 2011-12-10 18:18 esri_pcs_srs.txt
-rw-r--r-- 1 even even     0 2011-12-10 18:18 esri_pcs_wkt.txt

Seems to be due to ofile_wkt file handle being only opened and closed, but never written

comment:4 Changed 5 years ago by etourigny

Fixed in r23511 .

Status returned is now 'expected_fail' when at least one SRS does not match. Note that these errors have not been fixed yet!

comment:5 Changed 5 years ago by etourigny

r23520 : manage special cases of PROJECTION parameters that have multiple mappings. This fixes many apparent errors due to multiple mappings of PROJECTION names, namely: Transverse_Mercator/Gauss_Kruger , Equidistant_Cylindrical/Plate_Carree , Hotine_Oblique_Mercator_Azimuth_Natural_Origin/Hotine_Oblique_Mercator_Azimuth_Center.

example:

ERROR: SRS not matching for # 53001
PROJCS["Sphere_Plate_Carree",GEOGCS["GCS_Sphere",DATUM["D_Sphere",SPHEROID["Sphere",6371000.0,0.0]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Plate_Carree"],PARAMETER["False_Easting",0.0],PARAMETER["False_Northing",0.0],PARAMETER["Central_Meridian",0.0],UNIT["Meter",1.0]]
PROJCS["Sphere_Plate_Carree",GEOGCS["GCS_Sphere",DATUM["D_Sphere",SPHEROID["Sphere",6371000.0,0.0]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.017453292519943295]],PROJECTION["Equidistant_Cylindrical"],PARAMETER["False_Easting",0.0],PARAMETER["False_Northing",0.0],PARAMETER["Central_Meridian",0.0],UNIT["Meter",1.0]]

r23521: remap node if new value is not empty OGR_SRSNode::applyRemapper now only applies the remapped value if it is not "". This fixes many bugs where the DATUM value was set to 'D_' after morphFromESRI and threre is no corresponding ESRI_DATUM_NAME entry in gdal_datum.csv. Ideally these known datums should be added to gdal_datum.csv for completeness.

example:

ERROR: SRS not matching for # 2989
PROJCS["RRAF_1991_UTM_20N",GEOGCS["GCS_RRAF_1991",DATUM["D_RRAF_1991",SPHEROID["WGS_1984",6378137.0,298.257223563]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Transverse_Mercator"],PARAMETER["False_Easting",500000.0],PARAMETER["False_Northing",0.0],PARAMETER["Central_Meridian",-63.0],PARAMETER["Scale_Factor",0.9996],PARAMETER["Latitude_Of_Origin",0.0],UNIT["Meter",1.0]]
PROJCS["RRAF_1991_UTM_20N",GEOGCS["GCS_RRAF_1991",DATUM["D_",SPHEROID["WGS_1984",6378137.0,298.257223563]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.017453292519943295]],PROJECTION["Transverse_Mercator"],PARAMETER["False_Easting",500000.0],PARAMETER["False_Northing",0.0],PARAMETER["Central_Meridian",-63.0],PARAMETER["Scale_Factor",0.9996],PARAMETER["Latitude_Of_Origin",0.0],UNIT["Meter",1.0]]

comment:6 Changed 5 years ago by etourigny

Cc: warmerdam added

r23522: fix ESRI_DATUM_NAME for D_MOLDREF99 and D_Philippine_Reference_System_1992.

2 Entries in gdal_datum.csv were erroneous.

D_Phillipine_Reference_System_1992 was a typo ( should be D_Philippine_Reference_System_1992).

According to http://help.arcgis.com/en/arcims/10.0/mainhelp/mergedProjects/ArcXMLGuide/elements/gcs.htm Moldref 99 uses D_MOLDREF99 and not D_International_1967:

4023	GCS_MOLDREF99
GEOGCS["GCS_MOLDREF99",DATUM["D_MOLDREF99",SPHEROID["GRS_1980",6378137.0,298.257222101]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]]

To Frank: do any changes in *.csv files have to be made elsewhere so they don't get overwritten by a future update of the EPSG database?

Also: There were a number of missing ESRI_DATUM_NAME entries that this test showed (masked by fix in r23521), should I also add them to gdal_datum.csv ?

comment:7 Changed 5 years ago by etourigny

r23524: add test OGC->ESRI->OGC from EPSG codes and cleanup ESRI->OGC->ESRI tests

The new test fails for 20 EPSG codes in gcs.csv and 109 codes in pcs.csv , due to differences in datum names when 1 ESRI datum name maps to many datums in gdal_datum.csv .

Perhaps gdal_datum.csv should be revised and only allow 1 instance per ESRI Datum name?

For Example, D_Bogota maps to 2 named datums:

in tmp/epsg_gcs_srs.txt:

ERROR: SRS not matching for # 4802
GEOGCS["Bogota 1975 (Bogota)",DATUM["Bogota_1975_Bogota",SPHEROID["International 1924",6378388,297]],PRIMEM["Bogota",-74.08091666666667],UNIT["degree",0.0174532925199433]]
GEOGCS["Bogota 1975 (Bogota)",DATUM["Bogota_1975",SPHEROID["International_1924",6378388,297]],PRIMEM["Bogota",-74.08091666666667],UNIT["Degree",0.017453292519943295]]

in gdal_datum.csv:

6218,Bogota 1975,geodetic,"Fundamental point: Bogota observatory. Latitude: 4°35'56.570""N, longitude: 74°04'51.300""W (of Greenwich).",1975,7022,8901,3686,Topographic mapping.,"Replaces 1951 adjustment. Replaced by MAGNA-SIRGAS (datum code 6685).","Instituto Geografico Agustin Codazzi (IGAC) special publication no. 1, 4th edition (1975) ""Geodesia: Resultados Definitvos de Parte de las Redes Geodesicas Establecidas en el Pais"".",OGP,"2010/03/30",2000.200 2005.060 2007.060 2008.045 2009.106,0,"D_Bogota"
6802,"Bogota 1975 (Bogota)",geodetic,"Fundamental point: Bogota observatory. Latitude: 4°35'56.570""N, longitude: 0°E (of Bogota).",1975,7022,8904,3229,Topographic mapping.,,,OGP,"2008/06/24",2000.200 2003.361 2008.045,0,"D_Bogota"

comment:8 Changed 5 years ago by Even Rouault

Not sure if Frank follows closely this ticket, but yes indeed gdal_datum.csv is refreshed during the process described in http://svn.osgeo.org/metacrs/geotiff/trunk/libgeotiff/csv/README

comment:9 Changed 5 years ago by etourigny

r23541 : manage special cases in osr_esri_test_ogc_esri_ogc (missing rectified_grid_angle or scale_factor)

This fixes a few apparent errors, which are due to differences in GDAL vs. ESRI projection parameters.

r23542 : add optional fixing of TOWGS84, DATUM and GEOGCS

This fixes problem described above, when ESRI uses the same datum name for 2 different EPSG named datums (e.g. D_Bogota => Bogota_1975_Bogota and Bogota_1975). For now functionality is not active by default, so created a new test with GDAL_FIX_ESRI_WKT=DATUM which throws no errors.

This patch also addresses issues in bug #4345).

comment:10 Changed 5 years ago by etourigny

r23558 : add tests for EPSG codes

Reveals quite a few missing ESRI_DATUM_NAME in gdal_datum.csv .

I generated a new gdal_datum.csv using a slightly modified add_esri_column.py and file esri_gcs.csv derived from gcs.htm .

When that new gdal_datum.csv is used, there are less errors (104 for test osr_esri_21 and 58 for test osr_esri_23)

Attaching the modified files for reference, gdal_datum.csv has less missing ESRI datums. Ideally more should be added to esri_datum_override.csv to fill all missing data (58 from osr_esri_23).

Changed 5 years ago by etourigny

Attachment: add_esri_column.py added

Changed 5 years ago by etourigny

Attachment: esri_datum_override.csv added

Changed 5 years ago by etourigny

Attachment: esri_gcs.csv added

Changed 5 years ago by etourigny

Attachment: gdal_datum.csv added

comment:11 Changed 3 years ago by Kyle Shannon

Cc: Kyle Shannon added
Note: See TracTickets for help on using tickets.