CS-Map RFC # - NSRS2007 / NSRS 2011 Implementation

This page contains an change request (RFC) for the CS-Map Open Source project. More CS-Map RFCs can be found on the RFCs page. The change described in this request is to add support for the relatively new National Spatial Reference Systems of 2007 and 2011. These are, essentially, horizontal and vertical geodetic reference systems for the United States and its territories; produce by the National Geodetic Service of the United states. The acronyms NSRS2007 and NSRS2011 are commonly used to refer to these systems. They are also known as NAD83(2007) and NAD83(2011).

Status

RFC Template Version	(1.0)
Submission Date	18 February 2014
Last Modified	18February 2014
Author	Norm Olsen
RFC Status	Draft
Implementation Status	Developed, under test
Proposed Milestone	Don't know who/how milestones are assigned; possibly 13.30(?)
Assigned PSC guide(s)	Norm Olsen(?)
Voting History
+1	Norm
+0
-0
-1
no vote

Overview

While National Spatial Reference System 2007 )NSRS2007) has been around for several years, the shift defined by the new system, relative to the previous system (NAD83/96, aka HARN, HPGN, NAD83/91; we'll use HARN in this RFC document) was considered to small to deserve a defined shift definition. That is, the shifts were on the order of a few centimeters and at that time this was considered to be as small as the level of error. Fats forward to 2013, and precise and definitive geodetic shift models have been developed for NSRS2007. This was done at the time the US National Geodetic Survey was defining the National Spatial Reference System of 2011. Thus, at the current time, definitive models and algorithms exist for the migration of geodetic coordinates from HARN to NSRS2007 and subsequently to NSRS 2011.

Motivation

Please note that this is a sequential conversion process. That is, geodetic coordinates referenced to NAD83 must first be converted to HARN, and then converted to NSRS2007, before they can finally be converted to NSRS2011. Each of these three conversion processes has its own level of error, and each produces shifts of centimeters in magnitude. The following table indicates the expected magnitude of shifts for the three distinct datum shift calculations:

Datum Shift Calculation	Expected Shift in centimeters
NAD83 --> HARN	40
HARN --> NSRS2007	3
NSRS2007 --> NSRS2011	3

Unlike the NAD83 <--> HARN shift which has now been available for almost 2 decades, the new references systems do include vertical components. The vertical shifts are of the same magnitude as the horizontal shift indicated above. Perhaps this vertical shift is important to some users.

Finally, the new reference system definitions include Alaska which was never included in the HARN series of datum shift files. Quite frankly, I'm unaware of exactly how to convert Alaskan geography from NAD83 to NSRS2007. NSRS models do exist for Puerto Rico and the Virgin Islands, but not for Hawaii or American Samoa for which HARN datum shift files have been defined.

So, the actual significance of this change is questionable. A small portion of the user base is likely to want these features. The implementation of these new geodetic systems no negative effect on the operation of the library other than, perhaps, disk space. The data files used to model the NSRS2007 and NSRS2011 shifts are relatively huge, and each of the three geographic areas covered (48 states, Alaska, and Puerto Rico) requires three distinct data files (longitude shift, latitude shift, and vertical shift). Each of the data files for the 48 states are about 28 megabytes in size. Thus, the size of distribution files could grow to be unreasonably large if full support of the NSRS2007 & NSRS2011 geodetic reference systems is to be included.

Please remember, you can't get to NSRS2011 without first converting to NSRS2007. So, including coverage for 49 states in three dimensions for both of the new reference systems in a distribution will require an additional 456 megabytes of data. It is indeed possible, and perhaps likely, that the data files used in these new reference systems could be de-densifed - a grid cell in all of these files is one minute by one minute; but that will have to be the subject of a new and different RFC.

Proposed Solution

From a technical aspect of view, implementing NSRS2007 and NSRS2011 is rather straight forward. The models for these datum shifts as produced by the NGS is based on a new file format/interpolation combination which the NGS refers to as GEOCON. This format is very similar (almost identical) to previous file formats used, is that used for both horizontal and vertical NSRS datum shifts, and is also now used for the latest geoid height models produced by NGS. Adding a new grid interpolation file format named GEOCON to the repertoire which understands this new file format is relative straight forward; several standard modules are coded and the appropriate entries made in the grid file interpolation table. These code changes can be examined by looking at CS_geocn.c and CSdataDT.c

Having made the GEOCON grid file format option available, the NSRS2007 and NSRS2011 geodetic transformations can be easily be defined in the Geodetic Transformation Dictionary by referencing the appropriate files. As Datum definitions are now simply name placeholders, the NSRS2007 and NSRS2011 datums are easy to define.

Having the above in place leaves the more difficult and error prone, bureaucratic portion of the implementation. Theis phase includes several tasks which are intended to be done - to the degree possible - via automated means. The process here typically implies:

1. writing a program which produces tabular information in 'C++' code syntax,
2. manually editing the result tabular data to add/remove exemplary cases, and
3. writing a program to manipulate data (such as a dictionary source file) using the modified tablular data.

The result is a modification to, for example, a dictionary file which can be examined and tested. Should test results indicate, the program/table is adjusted and the modifictaion process is repeated until the modified data file (typiocally a dictionary source file) satifies the necessary requirements. In this manner, the modifications to the necessary data files are accomplished with the lowest possible probability of error, typographical or otherwise. These various programs and reulsting tables are all version controlled in the folder named ConsoleUtilities.

Using the abovev described technique, the following chores must be accomplished:

• A NSRS2007 version of most all HARN CRS systems must be generated.
• A NSRS2011 version of most all HARN CRS systems must be generated.
• Catalog entries for all new CRS systems must be generated.
• NameMapper entries for all new systems must be generated.
• Entries for the EPSG names and EPSG code must be added to the NameMapper and properly associated with the new CS-MAP NSRS CRS definition.
• Extries for the ESRI names and ESRI codes must be added to the NameMapper and properly associated with the new CS-MAP NSRS CRS definitions and the EPSG Names and codes.
• To perform that immediately above, programs which manipulate ESRI projection files (i.e. the WKT files with the .prj extension) need to written to properly extract ESRI names and code.
• Programs which will properly match EPSG, ESRI, and CS-MAP definition names and codes need to be written.

Having accomplished all of the above, NSRS2007 and NSRS 2011 will have been implemented, but with one major issue remaining.

The HPGN Problem

As indicated above, conversion from NAD83 to NSRS2007 first requires a conversion to HARN (aka NAD83/96, HPGN, NAD83/91, etc.). Conversion from NAD83 to HARN is accomplished through the use of grid interpolation files with names in the form of ??hpgn.l?s where the first two characters are generally replaced with a two characetr state code. For example, two data files named cohpgn.las and cohpgn.los define the shift required to convert from NAD83 to HARN in the state of Colorado. The .las file carries the latitude shift values, while the .los file carries the longitude shift values. The cmplicating factor here is that these data file naturally overlap and the shifts carries in the overlapping files are not the same in the regions of overlap. Thus, for geography covered by the regions of overlap there are indeed two (or more) officially designated shift values.

This is why in release 13.0 of CS-MAP, the original HARN datum was replaced by some 40+ distinct datums named HARN/??. Each of these new datums essentially provided the end user with the means to address a very specific set of HPGN data files. This capability was greeted with much appreciation for those working geography in the regions of HPGN file overlaps.

With the introduction of NSRS2007, however, we now have yet another problem. At the datum shoft calculation level, we only have geographic coordinates to work with. If such geographic coordinates came from the inverse projection conversion of a state plane CRS is unknown at this level. Thus, the conversion of, say, NAD83 to NSRS2007 presents the problem of which specific set of HPGN data files should be used for any given coordinate. There is no clear, unambiguous solution to this problem.

The 48hpgn.l?s File

It is proposed that this problem be solved by the generation of a new totally unofficial set of HPGN files which models the NAD83 <--> HARN datum shift for all 48 conterminous states. This file to be produced using the followoing concepts:

• All HPGN daum shift files have a standard grid cell size of 15 minutes of latitude/longitude.
• Fairly definitive state boundaries are available from teh EPSG database; certainly definitive enough to predict in which state any 15 minute Lat/long coordinate resides in.
• We then establish a HPGN grid file set named 48hpgn.las and 48hpgn.los which covers the entirity of the 48 conterminous states and has a 15 minute grid cell size.
• For each data point in the 48HPGN dta file set:
  • Determine the state in which the point lies
  • Determine the specific HPGN data file set from which values are to be extracted,
  • Extract the appropriate data values and insert into the 48HPGN data file set being created.

This process will succedd in generating valid coverage for most all of the 48 state geography. Problems occur where there are substantial waterways between states which are not included in the state boundary polygons; CHesapeak Bay, and the Great Lakes for example. These situation need to be detected and than manually dealt with.

Additional Complications

The following information will be of value to anyone reviewing the proposed changes:

1. There are three cases where the official HPGN data files set cover more than one state:
   • the wohpgn.l?s data file set covers both Washington and Oregon
   • the nehpgn.l?s data file set covers VT, NH, MA, and CT; i.e. New England (sans Maine)
   • the mdhpgn.l?s data file set convers both Maryland and Deleware.
   • the hihpgn.l?s files are for Hawaii, but don't overlapo anything, so there is no problem.

2> There are three states for which there are two HPGN data file sets:

• The cshpgn.l?s and cnhpgn.l?s files cover southern and northern California repectively.
• the emhpgn.l?s and wmhpgn.l?s files cover eastern and western Montana respectively.
• the ethpgn.l?s and wthpgn.l?s files cover eastern and western Texas respectively.
• the eshpgn.l?s and wshpgn.l?s files cover eastern and western Ammerican Samoa repectively. These files oonly overlap with themselves, and the region of overlap is all Pacific Ocean, so these file do not present a problem.

Of course, in the case of multiple HPGN data file sets covering a single state the files overlap, and yes the grid shift values in the regions of overlap are not the same. It is now known, for example, in the region of overlap in Montana, the differences between the two files is as much as 17 centimeters.

Result Evaluation

In regions internal to a specific state, the 48HPGN file solution produces results basically identical with the previous releases. The exact same interpolation code is used on the same data values. However, when one approaches the the boundary of any specific HPGN region (i.e. thegeography where HPGN data file overlap) the results can be as much as 10 centimeters different.

The best example of this condition is the four corneres area of the southwestern United States, a point of geography where four state borders come together and there are four different sets of HPGN data files which cover the intersection point. Thus there are four separate values of the shift at that point. The Colorado and Uath numbers, while not identical, are very close. The Arizona and New Mexico numbers are very close. Howevere, there is a 10 centimeter difference between the two sets of numbers. Who is to say which is correct?

Testing

Testing of the horizontal shifting at the geodetic coordinate level (i.e. lat/longs) will be accomplished using data points generated by the NGS supplied executable geocon.exe. This program accepts input data, and produces output data, in what is known as Blue Book format. In order to facilitate the generation of test data in the form used by the CS-MAP ConsoleTestCpp program, a TcsBlueBook object has been developed and program which produce test point data in this format, and the eventually convert the Blue Book data produced by geocon.exe into the TEST.DAT form have also been written.

Generally, the results match at the +/- 2 millimeter level. Precision is difficult to maintain in this testing environment as the Blue Book format only supports five didgits of precision at the arc-second level. Thus, verifying results at that low precision value get into trying to make sure ASCII to real and real to ASCII conversions, along with conversions from/to degrees, minutes, and seconds are now all part of the verification issue.

Tests generated for inclusion in the standard TEST.DAT file used by ConsoleTestCpp do not test vertical components (a weakness which deserves attention). Thus, testing the vertical component of the NSRS2007/NSRS2011 implementation needs to be done manually.

Implications

The implementation as is currently being tested is maintain in the CS-MAP Subversion repository as a sandbox named NSRS2007.

This implementation leaves all of the precious HARN/?? datum shifts in place. It introduces a new datum named NAD83/HARN. The transformations referneced to this new datum are the only transformations which reference the new 48HPGN data set. End users which have the need to have a specific set of HPGN grid shift files can achieve this by simply placing an entry for the superceeding HPGN data file set ahead of the 48HPGN data file set reference in the Geodetic Transformation Dictionary. Thus, a user working in the Four Corners area of the southwest, can specificy that, for example, the Arizona datum shift values are to prevail can do so by entering a reference to azhpgn,l?s ahead of the reference to 48hpgn.l?s. Editing the Geodtic Transformation Dictionary is recognized as not being an ideal solution, but given the nature of the beast, it is the best available short of adding the possibility of user interaction required when certain coordinate conversions are selected.

Funding/Resources

The implementation is complete and resides in a CS-MAP Subversion sandbox named NSRS2007. Additional test cases will need to added, and the current ConsoleTestCpp TEST.DAT file does not support vertical testing; thus vertical testing will need to be done separately. Additionally, x64 and Linux testing is yet to be performed.

Funding was provided by Autodesk. Upon approval of this RFC, the proposed changes will be submitted to the trunk.