Changes between Version 33 and Version 34 of NetCDF_exportUpdateChanges


Ignore:
Timestamp:
Oct 12, 2011, 11:23:11 PM (13 years ago)
Author:
pds
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • NetCDF_exportUpdateChanges

    v33 v34  
    1010
    1111The changes are listed here for comment:- including noting where applicable when they may affect backwards-compatibility with existing data exported by GDAL into NetCDF format.
    12 == Rationale ==
     12
     13== Overall Rationale ==
    1314
    1415The common rationale underlying these changes is:
    … …  
    3132== Planned Changes ==
    3233
    33 === Changes to metadata items including update to CF-1.5 ===
     34=== 1) Changes to metadata items including update to CF-1.5 ===
    3435
    3536As is common with many programs which manipulate netcdf data, a new GDAL metadata item (with version number) is added.  Also, the netcdf history attribute will be updated, and the Conventions attribute will be set to CF-1.5 to reflect support of the latest CF conventions and projections (See http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.5/).
    … …  
    4344'''Has back-compat impact?''': No
    4445
    45 === Change to a default south-up orientation for NetCDF data ===
     46=== 2) Change to a default south-up orientation for NetCDF data ===
    4647
    4748'''Related fix''': #4251[[BR]]
    … …  
    6061CF-1's default orientation is south-up, and all known current CF-1 -compliant applications expect this.
    6162
    62 
    63 === SRS Handling: Continue saving GDAL custom attribute tags by default to CRS definition, but add option not to save ===
     63=== 3) Changes to improve SRS Interoperability between GDAL and NetCDF CF-1 for Map Projected data ===
     64
     65See GDAL ticket: #2893
     66
     67While GDAL uses the OGC Well Known Text (WKT) format internally to store datasets' Spatial Reference Systems (SRS), the CF-1 conventions use a custom system based on a special 'grid_mapping' NetCDF variable with a set of projection attributes, and also Coordinate variables for both the projected coordinates and latitude and longitude. See [[http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.5/ch05s06.html CF1.5 ch5.6]] for an explanation and examples, and [[http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.5/apf.html CF1.5 App F]] for a list of supported projections.
     68
     69Unfortunately, the CF-1 conventions do not currently allow saving of all data needed for a fully georeferenced SRS, such as Named datums. For background on this, see CF convention trac tickets [[https://cf-pcmdi.llnl.gov/trac/ticket/9 #9]], [[https://cf-pcmdi.llnl.gov/trac/ticket/18 #18]] and [[https://cf-pcmdi.llnl.gov/trac/ticket/27 #27]].
     70
     71The proposed changes in this section update the NetCDF driver when exporting to NetCDF to:
     72 * save extra information so datasets are CF-1 compliant and can be viewed as gridded data in CF-1 compliant NetCDF applications built on NetCDF Java (X/Y projected coordinate arrays, and optionally full Lat/Lon arrays);
     73 * save as much of the SRS information in CF-1 compliant format as possible (including ellipsoid parameters);
     74 * continue exporting extra GDAL information by default that allows full re-import. However add to the driver a
     75   creation option so the user can choose not to export them.
     76
     77The following subsections describe these changes specifically.
     78
     79==== 3.1) Proj. SRS Handling: When exporting to NetCDF map projections via GDAL, save projection coordinate variables. Optionally, also save lat and lon mapping variables. ====
     80
     81'''Related fix''': #2893 [[BR]]
     82'''New -co options''':
     83  * WRITE_LONLAT=YES/NO/IFNEEDED (default: YES for geographic, IFNEEDED for projected) ->Note, this not implemented yet
     84  * TYPE_LONLAT=float/double (default: double for geographic, float for projected)
     85
     86'''Has back-compat impact?''': No
     87
     88'''Change''':
     89
     90When exporting rasters to NetCDF, save Coordinate Variables for the projected SRS, as required by the CF-1 convention, normally saved as "x" and "y".
     91
     92Optionally using -co options listed above, also write 2D 'lat' and 'lon' arrays, which are specified as part of the CF-1 convention for map projections - but are not actually required for the Unidata-maintained NetCDF Java API and applications built upon them (eg ncWMS, Thredds Data Server).
     93
     94The 'IFNEEDED' option of WRITELONLAT is used in the case of exporting a projection which is not supported by the CF-1.5 standard (e.g. sinusoidal), so any CF-1.5 compliant software can locate the points on a lon/lat map.
     95
     96'''Rationale''':
     97
     98The CF-1 conventions specify these coordinate variables as necessary for grids, see [[http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.5/ch05s06.html ch5.6]] and [[http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.5/apf.html App F]] in the document.
     99
     100Tests with NetCDF CF-1 reading software indeed show that rasters with projected grids will not be correctly interpreted without the coordinate variables in the projected system (the NetCDF-Java API determines this, since it's used by ncWMS, ToolsUI and Thredds Data Server).
     101
     102While the CF-1 convention examples also contain 2D "lat" and "lon" Variables containing the latitude and longitude at each data point, tests have shown that the NetCDF Java API can correctly interpret CF-1 data without these added. The rationale for this point is given at http://www.mail-archive.com/cf-metadata@cgd.ucar.edu/msg00759.html.
     103
     104Thus we've chosen to make it ''optional'' to write these Variables, as they are redundant from GDAL's point of view (Lat and Lon are calculated on-the-fly), and they increase the size of a NetCDF file significantly. (E.g. if the actual band-data is stored as a 2D Byte array, but the lat and lon are stored as doubles, then the file-size can easily increase by 10X).
     105
     106==== 3.2) Proj. SRS Handling: Add saving of reference Ellipsoid parameters in CF-1 compliant format used by dataset ====
     107
     108'''Related fix''': #2893 [[BR]]
     109'''New -co options''': No, though related to the new proposed WRITE_GDAL_TAGS option above.
     110
     111'''Has back-compat impact?''': No
     112
     113'''Change''':
     114
     115When exporting to NetCDF, the new driver will utilise the CF-1 attributes that allow specification of the reference ellipsoid used by the dataset's CRS (e.g. 'semi_major_axis', 'inverse_flattening').
     116
     117However, on importing data the driver will continue to use the full GDAL WKT specification in the 'spatial_ref' attribute where present. This is because the CF-1.5 conventions still don't include provision for saving full Datum information, such as the name and authority of the datum, which is needed for precise transformations. The CF-1 ellipsoid attributes will only be used as a fallback when the full WKT isn't present.
     118
     119{{{
     120#!comment
     121((?? pds: Should we add anything here about special import behaviour, e.g. whether to auto-detect datum on import based on ellipsoid values?? Or should there be an option for this?))
     122
     123-> etienne: we could look into that for most common datums, but specify that there is no guarantee that a specific datum will be checked for.
     124}}}
     125
     126'''Rationale''':
     127
     128CF-1 convention's only support of datums currently is to specify a reference ellipsoid using parameters such as 'semi_major_axis' and 'inverse_flattening'.
     129
     130While these attributes alone do not fully specify the datum (which requires an EPSG code so control points can be referenced), they at least allow fairly accurate map projection of the created NetCDF file using tools such as ncWMS.
     131
     132In the longer-term it would be beneficial to work with the CF-1 community to allow saving of full datum information in a format readable by CF-1.
     133
     134==== 3.3) Proj. SRS Handling: Continue saving GDAL custom attribute tags by default to CRS definition, but add option not to save ====
    64135
    65136'''Related fix''': N/A [[BR]]
    … …  
    81152
    82153{{{
    83 !comment
     154#!comment
    84155Note: when importing files in NetCDF format, the driver checks the GDAL-written WKT matches the projection information stored in the CF-compliant attributes. If there is a conflict, the CF-compliant CRS will be considered as authoritative.
    85156}}}
    … …  
    93164Note: In the longer-term we would view it as advantageous that the CF-1 conventions were augmented to allow recording of full datum information, and reduce the need to store extra CRS information in GDAL-managed attributes. However this will require agreement of the CF-1 community for a new standard, and would take time to agree upon and adopt.  See https://cf-pcmdi.llnl.gov/trac/ticket/69 .
    94165
    95 === SRS Handling: Add saving of reference Ellipsoid parameters in CF-1 compliant format used by dataset ===
    96 
    97 '''Related fix''': #2893 [[BR]]
    98 '''New -co options''': No, though related to the new proposed WRITE_GDAL_TAGS option above.
    99 
    100 '''Has back-compat impact?''': No
    101 
    102 '''Change''':
    103 
    104 When exporting to NetCDF, the new driver will utilise the CF-1 attributes that allow specification of the reference ellipsoid used by the dataset's CRS (e.g. 'semi_major_axis', 'inverse_flattening').
    105 
    106 However, on importing data the driver will continue to use the full GDAL WKT specification in the 'spatial_ref' attribute where present. This is because the CF-1.5 conventions still don't include provision for saving full Datum information, such as the name and authority of the datum, which is needed for precise transformations. The CF-1 ellipsoid attributes will only be used as a fallback when the full WKT isn't present.
    107 
    108 {{{
    109 #!comment
    110 ((?? pds: Should we add anything here about special import behaviour, e.g. whether to auto-detect datum on import based on ellipsoid values?? Or should there be an option for this?))
    111 
    112 -> etienne: we could look into that for most common datums, but specify that there is no guarantee that a specific datum will be checked for.
    113 }}}
    114 
    115 '''Rationale''':
    116 
    117 CF-1 convention's only support of datums currently is to specify a reference ellipsoid using parameters such as 'semi_major_axis' and 'inverse_flattening'.
    118 
    119 While these attributes alone do not fully specify the datum (which requires an EPSG code so control points can be referenced), they at least allow fairly accurate map projection of the created NetCDF file using tools such as ncWMS.
    120 
    121 In the longer-term it would be beneficial to work with the CF-1 community to allow saving of full datum information in a format readable by CF-1.
    122 
    123 === SRS Handling: When exporting to NetCDF map projections via GDAL, save projection coordinate variables. Optionally, also save lat and lon mapping variables. ===
    124 
    125 '''Related fix''': #2893 [[BR]]
    126 '''New -co options''':
    127   * WRITE_LONLAT=YES/NO/IFNEEDED (default: YES for geographic, IFNEEDED for projected) ->Note, this not implemented yet
    128   * TYPE_LONLAT=float/double (default: double for geographic, float for projected)
    129 
    130 '''Has back-compat impact?''': No
    131 
    132 '''Change''':
    133 
    134 When exporting rasters to NetCDF, save Coordinate Variables for the projected SRS, as required by the CF-1 convention, normally saved as "x" and "y".
    135 
    136 Optionally using -co options listed above, also write 2D 'lat' and 'lon' arrays, which are specified as part of the CF-1 convention for map projections - but are not actually required for the Unidata-maintained NetCDF Java API and applications built upon them (eg ncWMS, Thredds Data Server).
    137 
    138 The ifneeded option of WRITELONLAT is used in the case of exporting a projection which is not supported by the CF-1.5 standard (e.g. sinusoidal), so any CF-1.5 compliant software can locate the points on a lon/lat map.
    139 
    140 '''Rationale''':
    141 
    142 The CF-1 conventions specify these coordinate variables as necessary for grids, see [[http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.5/ch05s06.html ch5.6]] and [[http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.5/apf.html App F]] in the document.
    143 
    144 Tests with NetCDF CF-1 reading software indeed show that rasters with projected grids will not be correctly interpreted without the coordinate variables in the projected system (the NetCDF-Java API determines this, since it's used by ncWMS, ToolsUI and Thredds Data Server).
    145 
    146 While the CF-1 convention examples also contain 2D "lat" and "lon" Variables containing the latitude and longitude at each data point, tests have shown that the NetCDF Java API can correctly interpret CF-1 data without these added. The rationale for this point is given at http://www.mail-archive.com/cf-metadata@cgd.ucar.edu/msg00759.html.
    147 
    148 Thus we've chosen to make it ''optional'' to write these Variables, as they are redundant from GDAL's point of view (Lat and Lon are calculated on-the-fly), and they increase the size of a NetCDF file significantly. (E.g. if the actual band-data is stored as a 2D Byte array, but the lat and lon are stored as doubles, then the file-size can easily increase by 10X).
    149 
    150 
    151 === Add support for different netcdf file types and compression ===
     166
     167=== 4) Add support for different netcdf file types and compression ===
    152168
    153169'''Related fix''': N/A [[BR]]