Changes between Version 10 and Version 11 of MapGuideRfc116


Ignore:
Timestamp:
Jun 29, 2011, 10:52:30 AM (13 years ago)
Author:
NormOlsen
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • MapGuideRfc116

    v10 v11  
    2323== Overview ==
    2424
    25 Geodetic coordinate system conversion is involved in a large portion of !MapGuide data processing at both the server and client levels.  Presuming the acceptance and implementation of !CS-MAP !RFC !#5, it is proposed that the !MgCoordinateSystem Application Programmers Interface (API) be enhanced to increase the performance of coordinate conversions at both the server and client levels of !MapGuide without changing the signature or behavior of any existing member of the API.
     25Geodetic coordinate system conversion is involved in a large portion of !MapGuide data processing at both the server and client levels.  Presuming the acceptance and implementation of CS-MAP RFC !#5, it is proposed that the !MgCoordinateSystem Application Programmers Interface (API) be enhanced to increase the performance of coordinate conversions at both the server and client levels of !MapGuide without changing the signature or behavior of any existing member of the API.
    2626
    2727== Motivation ==
    2828
    29 Regardless of how fast anything on the Internet is, it is never fast enough.  Hardware available at both the server and client level is now multi-core capable thus enabling higher performance.  Software developments such as cloud computing provide the demand for higher performance.  The changes proposed in this !RFC are intended to address these issues.
     29Regardless of how fast anything on the Internet is, it is never fast enough.  Hardware available at both the server and client level is now multi-core capable thus enabling higher performance.  Software developments such as cloud computing provide the demand for higher performance.  The changes proposed in this RFC are intended to address these issues.
    3030
    3131== Proposed Solution ==
    3232
    33 The proposed submission would not change any existing method signatures or change any behavior in a substantial way.  The proposal introduces five new members to the currently existing !MgCoordinateSystemTransform interface.  This !RFC includes an outline of the recommended usage of the !API which will provide the optimum performance of the !API.
     33The proposed submission would not change any existing method signatures or change any behavior in a substantial way.  The proposal introduces five new members to the currently existing !MgCoordinateSystemTransform interface.  This RFC includes an outline of the recommended usage of the API which will provide the optimum performance of the API.
    3434
    35 Using the metric of the number of conversions per second from !UTM27-13 to !CO83-C in a pure measurement environment (i.e. no coordinate retrieval or delivery code), the underlying !CS-MAP library is capable of producing approximately 1 million conversions per second on an average desktop machine.  Changes in the !MapGuide !API, therefore, cannot get us beyond this limit.  Thus, in this !RFC, we will write of performance in terms of the percentage of this theoretical maximum which the !API can/will deliver.  The current implementation of the !API delivers performance of approximately 80% of this maximum.  Research and test implementations indicate that it is not unreasonable to expect an improvement to 91% of the theoretical maximum when using the most efficient of the Transform function overloads.
     35Using the metric of the number of conversions per second from UTM27-13 to CO83-C in a pure measurement environment (i.e. no coordinate retrieval or delivery code), the underlying CS-MAP library is capable of producing approximately 1 million conversions per second on an average desktop machine.  Changes in the !MapGuide API, therefore, cannot get us beyond this limit.  Thus, in this RFC, we will write of performance in terms of the percentage of this theoretical maximum which the API can/will deliver.  The current implementation of the API delivers performance of approximately 80% of this maximum.  Research and test implementations indicate that it is not unreasonable to expect an improvement to 91% of the theoretical maximum when using the most efficient of the Transform function overloads.
    3636
    3737Achieving this level of improvement is deemed possible by four distinct tasks involving the API.
     
    3939=== Removing the Requirement for a Critical Section ===
    4040
    41 Currently, there are specific transformations within the CS-MAP library which are not threadsafe.  Therefore, to insure proper operation in a multi-threaded environment, a critical section is activated for all datum shift calculations.  It is the intent of this !RFC to remove this requirement.  Assuming the acceptance and implementation of !OsGeo !MetaCRS !RFC !#5, !CS-MAP will enable the !API to query !CS-MAP and determine if a critical section is necessary for a specific transformation.  The !API will be modified to use this information and invoke the critical section only as necessary.
     41Currently, there are specific transformations within the CS-MAP library which are not threadsafe.  Therefore, to insure proper operation in a multi-threaded environment, a critical section is activated for all datum shift calculations.  It is the intent of this !RFC to remove this requirement.  Assuming the acceptance and implementation of !OsGeo MetaCRS RFC !#5, CS-MAP will enable the API to query CS-MAP and determine if a critical section is necessary for a specific transformation.  The API will be modified to use this information and invoke the critical section only as necessary.
    4242
    43 Please note that the referenced !CS-MAP !RFC includes two phases.  The first phase includes only the identification of those transformations which are known to be reentrant and those which are not.  The first phase is a relatively simple task which is easily completed with available resources.  The second phase includes some, perhaps non-trivial, efforts to make threadsafe most, if not all, !CS_MAP conversions and transformations which are known to be non-threadsafe.  Thus, as this work progresses, further performance enhancements will inure to applications without and code changes required.
     43Please note that the referenced CS-MAP RFC includes two phases.  The first phase includes only the identification of those transformations which are known to be reentrant and those which are not.  The first phase is a relatively simple task which is easily completed with available resources.  The second phase includes some, perhaps non-trivial, efforts to make threadsafe most, if not all, CS-MAP conversions and transformations which are known to be non-threadsafe.  Thus, as this work progresses, further performance enhancements will inure to applications without and code changes required.
    4444
    4545=== Refactor the MgCoordinateSystemTransform::Transform Functions ===
    4646
    47 It is proposed that the existing implementation of all of the Transform overloads in the existing !MgCoordinateSystemTransform object be refactored for optimum performance purposes.  It is contemplated that by: a) reducing the number of changes the form of a coordinate takes, b) removing some internal function calls by replicating code to a small degree, and c) reducing the overhead implied by several layers of try {} catch blocks; that significant performance enhancements can be achieved.  This work will introduce some minor changes in behavior which are considered to be improvements in consistency and usefulness of the !API and only affects behavior in extraordinary cases.  These changes are detailed below.
     47It is proposed that the existing implementation of all of the Transform overloads in the existing !MgCoordinateSystemTransform object be refactored for optimum performance purposes.  It is contemplated that by: a) reducing the number of changes the form of a coordinate takes, b) removing some internal function calls by replicating code to a small degree, and c) reducing the overhead implied by several layers of try {} catch blocks; that significant performance enhancements can be achieved.  This work will introduce some minor changes in behavior which are considered to be improvements in consistency and usefulness of the API and only affects behavior in extraordinary cases.  These changes are detailed below.
    4848
    4949=== Conversion Status Accumulation ===
    5050
    51 !CS_MAP issues warnings for coordinates outside the useful range of the coordinate systems (and the datums referenced by them) used to construct the !MgCoordinateSystemTransform object.  These are warnings and do not mean that the returned coordinates are invalid.  It should '''not''' be considered abnormal for a small sub-set of the coordinates in a large conversion to be outside the useful range of a Transformation object.  In the event that a large number of coordinates in a conversion are found to be outside the useful range, it is proper to question the validity of the conversion.  Such a case is a strong indication that the user may not have selected the proper coordinate system for a specific conversion.
     51CS-MAP issues warnings for coordinates outside the useful range of the coordinate systems (and the datums referenced by them) used to construct the !MgCoordinateSystemTransform object.  These are warnings and do not mean that the returned coordinates are invalid.  It should '''not''' be considered abnormal for a small sub-set of the coordinates in a large conversion to be outside the useful range of a Transformation object.  In the event that a large number of coordinates in a conversion are found to be outside the useful range, it is proper to question the validity of the conversion.  Such a case is a strong indication that the user may not have selected the proper coordinate system for a specific conversion.
    5252
    53 The default behavior of the !API is to throw an exception whenever such a warning is returned by the !CS-MAP library.  This default behavior can be, and often is, modified at run-time using the !IgnoreDatumShiftWarning and !IgnoreOutsideDomainWarning members of the !MgCoordinateSystemTransform interface.  Thus, it is recommended that applications using the API disable the exception throwing behavior of the API.  It is further proposed that the !MgCoordinateSystemTransform object be enhanced to provide a status accumulation feature.  By status accumulation, we refer to the concept of: a)counting all source projective CRS warnings issued, b)counting all datum shift warnings issued, and c)counting all target projective CRS warnings.
     53The default behavior of the API is to throw an exception whenever such a warning is returned by the CS-MAP library.  This default behavior can be, and often is, modified at run-time using the !IgnoreDatumShiftWarning and !IgnoreOutsideDomainWarning members of the !MgCoordinateSystemTransform interface.  Thus, it is recommended that applications using the API disable the exception throwing behavior of the API.  It is further proposed that the !MgCoordinateSystemTransform object be enhanced to provide a status accumulation feature.  By status accumulation, we refer to the concept of: a)counting all source projective CRS warnings issued, b)counting all datum shift warnings issued, and c)counting all target projective CRS warnings.
    5454
    5555Upon construction, or upon use of the !SetSourceAndTarget member function, or upon use of the !ResetLastTransformStatus member function, all counters of the status accumulation mechanism will be reset to zero.  Each point converted by the !MgCoordinateSystemTransform object will cause the appropriate counts to be advanced based on the status of the conversion.  Upon completion of the conversion of a map or data source, the application would then query the Transform object and make a determination as to the validity of the result.
    5656
    57 For example, a conversion where the target !CRS warning count exceeds, say, 20% of the total number of points suggests that the target !CRS chosen by the user is inappropriate for the data set being converted.  On the other hand, warning counts which are less than, say, 20% of the total point count suggest a normal conversion.
     57For example, a conversion where the target CRS warning count exceeds, say, 20% of the total number of points suggests that the target CRS chosen by the user is inappropriate for the data set being converted.  On the other hand, warning counts which are less than, say, 20% of the total point count suggest a normal conversion.
    5858
    59 Thus, the following additional member functions to the MgCoordinateSystemTransform object are proposed:
     59Thus, the following additional member functions to the !MgCoordinateSystemTransform object are proposed:
    6060
    6161{{{ INT32 MgCoordinateSystemTransform::GetSourceWarningCount (void); }}}
     
    9494=== Critical Section Still Required ===
    9595
    96 It would be nice to assume that all current CS-MAP coordinate conversion algorithms can be made threadsafe without a serious affect on resources and/or performance, and that all future additions to the !CS-MAP library will be implemented in a threadsafe manner.  However, the ability to have a non-threadsafe conversion/transformation method in the !CS-MAP library is reserved.  Thus, we retain the Critical Section to keep multiple threads from using a non-threadsafe conversion or transformation at the same time.  Given the implementation of !CS-MAP !RFC !#5, however, we will only need to actually use it when truly necessary.
     96It would be nice to assume that all current CS-MAP coordinate conversion algorithms can be made threadsafe without a serious affect on resources and/or performance, and that all future additions to the CS-MAP library will be implemented in a threadsafe manner.  However, the ability to have a non-threadsafe conversion/transformation method in the CS-MAP library is reserved.  Thus, we retain the Critical Section to keep multiple threads from using a non-threadsafe conversion or transformation at the same time.  Given the implementation of CS-MAP RFC !#5, however, we will only need to actually use it when truly necessary.
    9797
    9898=== Thread Saftey ===
     
    105105
    106106A substantial portion of the increased performance to be achieved will be derived from a refactoring of the coordinate system conversion code.  Over the years, this code has become somewhat inefficient using several nested function calls with non-trivial signatures.  In refactoring this code, the following changes in behavior (more like corrections) will be made:
    107  1. In the existing code, the behavior of the !API with regard to the status of results in the event of an exception being thrown is inconsistent.  In the proposed code, conversion results will always be provided, even in the event of an exception being thrown.  Thus, the proposed behavior will provide consistent return results and also contribute to higher performance levels.  That is, even in the event of an exception, all coordinates requested to be converted will have been converted.
    108  2. The four status values returned in the !m_nTransformStatus member of the !MgCoordinateSystemTransform object will be adjusted to form a severity level sequence which rates a geodetic datum “outside range” as more severe than a projected “outside range”.  The names used will not change, only the numeric values assigned to them; so this should not require any coding changes.
    109  3. The overloads of the !MgCoordinateSystemTransform::Transform which deal with arrays will now always complete the conversion of the entire array before throwing any exception with regard to warning status values encountered in the conversion.  Also, these overloads will be modified so that the value of the !m_nTransformStatus member, upon return, will always reflect the worst status encountered (per the severity level described in 2 above) in the transformation of the array (as opposed to the status of the last conversion performed as is currently done).
    110  4. All overloads of the !TransformM variety will now always calculate and return the ‘m’ value.  Currently, when an exception is thrown, the XYZ coordinate values would be converted, but the ‘m’ value would not always be.
     107 1. In the existing code, the behavior of the API with regard to the status of results in the event of an exception being thrown is inconsistent.  In the proposed code, conversion results will always be provided, even in the event of an exception being thrown.  Thus, the proposed behavior will provide consistent return results and also contribute to higher performance levels.  That is, even in the event of an exception, all coordinates requested to be converted will have been converted.
     108 2. The four status values returned in the m_nTransformStatus member of the !MgCoordinateSystemTransform object will be adjusted to form a severity level sequence which rates a geodetic datum “outside range” as more severe than a projected “outside range”.  The names used will not change, only the numeric values assigned to them; so this should not require any coding changes.
     109 3. The overloads of the MgCoordinateSystemTransform::Transform which deal with arrays will now always complete the conversion of the entire array before throwing any exception with regard to warning status values encountered in the conversion.  Also, these overloads will be modified so that the value of the m_nTransformStatus member, upon return, will always reflect the worst status encountered (per the severity level described in 2 above) in the transformation of the array (as opposed to the status of the last conversion performed as is currently done).
     110 4. All overloads of the TransformM variety will now always calculate and return the ‘m’ value.  Currently, when an exception is thrown, the XYZ coordinate values would be converted, but the ‘m’ value would not always be.
    111111
    112 Coordinate results provided in the case of an exception will be what !CS-MAP considers to be a "rational result".  In the case of a datum shift calculation failure, the rational result is either that calculated by the fallback specification or the unshifted input coordinates (if there is no fallback).  This is considered rational as datums shifts are rarely more than 100 meters, and usually in the range of 20 meters.  Thus, given that the input coordinate is outside the useful range of the datum shift transformation (typically this means outside the coverage provided by grid shift data files), the result "rational result" is the unshifted input.
     112Coordinate results provided in the case of an exception will be what CS-MAP considers to be a "rational result".  In the case of a datum shift calculation failure, the rational result is either that calculated by the fallback specification or the unshifted input coordinates (if there is no fallback).  This is considered rational as datums shifts are rarely more than 100 meters, and usually in the range of 20 meters.  Thus, given that the input coordinate is outside the useful range of the datum shift transformation (typically this means outside the coverage provided by grid shift data files), the result "rational result" is the unshifted input.
    113113
    114 In the case of projective conversions, the "rational result" is based on the nature of the projection. For many of the projections supported, the "rational result" is simply what the projection mathematics produce, even though the coordinate is known to be outside the region for which the projection's parameters suggest is the useful range of the conversion.  In other cases, the projection will have singularity points, such as either pole in the case of the traditional !Mercator.  In such cases the "rational result" typically includes one or more ordinates with an unmistakably large number which suggests infinity, but will '''not''' cause a floating point exception if the value is used for any normal calculation.
     114In the case of projective conversions, the "rational result" is based on the nature of the projection. For many of the projections supported, the "rational result" is simply what the projection mathematics produce, even though the coordinate is known to be outside the region for which the projection's parameters suggest is the useful range of the conversion.  In other cases, the projection will have singularity points, such as either pole in the case of the traditional Mercator.  In such cases the "rational result" typically includes one or more ordinates with an unmistakably large value which suggests infinity, but will '''not''' cause a floating point exception if the value is used for any normal calculation.
    115115
    116116== Test Plan ==