Find and fix errors in ESRI WKT morphing

[etourigny]

ESRI WKT morphing magic needs some improvement and testing with actual data. Many ESRI EKT .prj files have been collected in #4345 .

Added simple autotest to osr/osr_esri.py and support data in r23510.

For each input ESRI WKT, the test morphs from and to ESRI and checks for matching SRS (with IsSame()) and also for matching WKT (non fatal as almost all fail this test due to precision differences).

Attaching first results, there are quite a few errors, mostly due to different or missing datums.

[etourigny]

log file with errors in PCS defifnitions

[etourigny]

Test files in osr/data have been obtained from ESRI, transformed to a CSV file (semi-colon separator) and gzipped for space. The file data/esri_extra.csv is to add any other definitions.

[Even Rouault]

data/esri_extra.csv has not been commited.

And I get 'fail' tests. Perhaps due to the fact that you have local changes in your ogr/ogr_srs_esri.cpp

  TEST: osr_esri_21 ... WARNING: Failed 738 WKT tests, see file tmp/esri_gcs_wkt.txt
fail
    line 824: ERROR: Failed 26 SRS tests, see file tmp/esri_gcs_srs.txt
  TEST: osr_esri_22 ... WARNING: Failed 4268 WKT tests, see file tmp/esri_pcs_wkt.txt
fail
    line 824: ERROR: Failed 48 SRS tests, see file tmp/esri_pcs_srs.txt
  TEST: osr_esri_23 ... fail
    line 767: input file data/esri_extra.csv does not exist

Test Script: osr_esri
Succeeded: 20
Failed:    3 (0 blew exceptions)
Skipped:   0
Expected fail:0

And the _wtk.txt files are always empty :

-rw-r--r-- 1 even even  8926 2011-12-10 18:18 esri_gcs_srs.txt
-rw-r--r-- 1 even even     0 2011-12-10 18:18 esri_gcs_wkt.txt
-rw-r--r-- 1 even even 42953 2011-12-10 18:18 esri_pcs_srs.txt
-rw-r--r-- 1 even even     0 2011-12-10 18:18 esri_pcs_wkt.txt

Seems to be due to ofile_wkt file handle being only opened and closed, but never written

[etourigny]

Fixed in r23511 .

Status returned is now 'expected_fail' when at least one SRS does not match. Note that these errors have not been fixed yet!

[etourigny]

r23520 : manage special cases of PROJECTION parameters that have multiple mappings. This fixes many apparent errors due to multiple mappings of PROJECTION names, namely: Transverse_Mercator/Gauss_Kruger , Equidistant_Cylindrical/Plate_Carree , Hotine_Oblique_Mercator_Azimuth_Natural_Origin/Hotine_Oblique_Mercator_Azimuth_Center.

example:

ERROR: SRS not matching for # 53001
PROJCS["Sphere_Plate_Carree",GEOGCS["GCS_Sphere",DATUM["D_Sphere",SPHEROID["Sphere",6371000.0,0.0]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Plate_Carree"],PARAMETER["False_Easting",0.0],PARAMETER["False_Northing",0.0],PARAMETER["Central_Meridian",0.0],UNIT["Meter",1.0]]
PROJCS["Sphere_Plate_Carree",GEOGCS["GCS_Sphere",DATUM["D_Sphere",SPHEROID["Sphere",6371000.0,0.0]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.017453292519943295]],PROJECTION["Equidistant_Cylindrical"],PARAMETER["False_Easting",0.0],PARAMETER["False_Northing",0.0],PARAMETER["Central_Meridian",0.0],UNIT["Meter",1.0]]

r23521: remap node if new value is not empty OGR_SRSNode::applyRemapper now only applies the remapped value if it is not "". This fixes many bugs where the DATUM value was set to 'D_' after morphFromESRI and threre is no corresponding ESRI_DATUM_NAME entry in gdal_datum.csv. Ideally these known datums should be added to gdal_datum.csv for completeness.

example:

ERROR: SRS not matching for # 2989
PROJCS["RRAF_1991_UTM_20N",GEOGCS["GCS_RRAF_1991",DATUM["D_RRAF_1991",SPHEROID["WGS_1984",6378137.0,298.257223563]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Transverse_Mercator"],PARAMETER["False_Easting",500000.0],PARAMETER["False_Northing",0.0],PARAMETER["Central_Meridian",-63.0],PARAMETER["Scale_Factor",0.9996],PARAMETER["Latitude_Of_Origin",0.0],UNIT["Meter",1.0]]
PROJCS["RRAF_1991_UTM_20N",GEOGCS["GCS_RRAF_1991",DATUM["D_",SPHEROID["WGS_1984",6378137.0,298.257223563]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.017453292519943295]],PROJECTION["Transverse_Mercator"],PARAMETER["False_Easting",500000.0],PARAMETER["False_Northing",0.0],PARAMETER["Central_Meridian",-63.0],PARAMETER["Scale_Factor",0.9996],PARAMETER["Latitude_Of_Origin",0.0],UNIT["Meter",1.0]]

[etourigny]

r23522: fix ESRI_DATUM_NAME for D_MOLDREF99 and D_Philippine_Reference_System_1992.

2 Entries in gdal_datum.csv were erroneous.

D_Phillipine_Reference_System_1992 was a typo ( should be D_Philippine_Reference_System_1992).

According to ​http://help.arcgis.com/en/arcims/10.0/mainhelp/mergedProjects/ArcXMLGuide/elements/gcs.htm Moldref 99 uses D_MOLDREF99 and not D_International_1967:

4023	GCS_MOLDREF99
GEOGCS["GCS_MOLDREF99",DATUM["D_MOLDREF99",SPHEROID["GRS_1980",6378137.0,298.257222101]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]]

To Frank: do any changes in *.csv files have to be made elsewhere so they don't get overwritten by a future update of the EPSG database?

Also: There were a number of missing ESRI_DATUM_NAME entries that this test showed (masked by fix in r23521), should I also add them to gdal_datum.csv ?

[etourigny]

r23524: add test OGC->ESRI->OGC from EPSG codes and cleanup ESRI->OGC->ESRI tests

The new test fails for 20 EPSG codes in gcs.csv and 109 codes in pcs.csv , due to differences in datum names when 1 ESRI datum name maps to many datums in gdal_datum.csv .

Perhaps gdal_datum.csv should be revised and only allow 1 instance per ESRI Datum name?

For Example, D_Bogota maps to 2 named datums:

in tmp/epsg_gcs_srs.txt:

ERROR: SRS not matching for # 4802
GEOGCS["Bogota 1975 (Bogota)",DATUM["Bogota_1975_Bogota",SPHEROID["International 1924",6378388,297]],PRIMEM["Bogota",-74.08091666666667],UNIT["degree",0.0174532925199433]]
GEOGCS["Bogota 1975 (Bogota)",DATUM["Bogota_1975",SPHEROID["International_1924",6378388,297]],PRIMEM["Bogota",-74.08091666666667],UNIT["Degree",0.017453292519943295]]

in gdal_datum.csv:

6218,Bogota 1975,geodetic,"Fundamental point: Bogota observatory. Latitude: 4°35'56.570""N, longitude: 74°04'51.300""W (of Greenwich).",1975,7022,8901,3686,Topographic mapping.,"Replaces 1951 adjustment. Replaced by MAGNA-SIRGAS (datum code 6685).","Instituto Geografico Agustin Codazzi (IGAC) special publication no. 1, 4th edition (1975) ""Geodesia: Resultados Definitvos de Parte de las Redes Geodesicas Establecidas en el Pais"".",OGP,"2010/03/30",2000.200 2005.060 2007.060 2008.045 2009.106,0,"D_Bogota"
6802,"Bogota 1975 (Bogota)",geodetic,"Fundamental point: Bogota observatory. Latitude: 4°35'56.570""N, longitude: 0°E (of Bogota).",1975,7022,8904,3229,Topographic mapping.,,,OGP,"2008/06/24",2000.200 2003.361 2008.045,0,"D_Bogota"

[Even Rouault]

Not sure if Frank follows closely this ticket, but yes indeed gdal_datum.csv is refreshed during the process described in ​http://svn.osgeo.org/metacrs/geotiff/trunk/libgeotiff/csv/README

[etourigny]

r23541 : manage special cases in osr_esri_test_ogc_esri_ogc (missing rectified_grid_angle or scale_factor)

This fixes a few apparent errors, which are due to differences in GDAL vs. ESRI projection parameters.

r23542 : add optional fixing of TOWGS84, DATUM and GEOGCS

This fixes problem described above, when ESRI uses the same datum name for 2 different EPSG named datums (e.g. D_Bogota => Bogota_1975_Bogota and Bogota_1975). For now functionality is not active by default, so created a new test with GDAL_FIX_ESRI_WKT=DATUM which throws no errors.

This patch also addresses issues in bug #4345).

[etourigny]

r23558 : add tests for EPSG codes

Reveals quite a few missing ESRI_DATUM_NAME in gdal_datum.csv .

I generated a new gdal_datum.csv using a slightly modified add_esri_column.py and file esri_gcs.csv derived from gcs.htm .

When that new gdal_datum.csv is used, there are less errors (104 for test osr_esri_21 and 58 for test osr_esri_23)

Attaching the modified files for reference, gdal_datum.csv has less missing ESRI datums. Ideally more should be added to esri_datum_override.csv to fill all missing data (58 from osr_esri_23).

[Even Rouault]

This ticket has been automatically closed because Trac is no longer used for GDAL bug tracking, since the project has migrated to GitHub. If you believe this ticket is still valid, you may file it to ​https://github.com/OSGeo/gdal/issues if it is not already reported there.