Update NetCDF driver for greater CF-1 convention compatibility (orientation, datums, map projections)

(Note: this is an initial draft by pds - to collect related ideas for input and if necessary RFC development)

Status: Development

Summary

In August 2011 Etienne Tourinsky initiated a discussion on the GDAL mailing list proposing and calling for interest in improvements to GDAL's NetCDF driver (Subject: discussion on improvements to the NetCDF driver and CF-1 convention), and collected ideas at wiki:NetCDF_Improvements.

This page summarises several of the proposed improvements that would improve the compatibility of GDAL's NetCDF driver with the CF-1 NetCDF conventions :- but change the default behaviour of the driver and thus affect backward compatibility.

The changes affect orientation of data, handling of datum parameters, writing coordinate variables, and are described specifically below.

Rationale

The common rationale underlying these changes is:

(1) The NetCDF view of rasters created by GDAL should conform to CF-1 conventions :- and work for key operations with common NetCDF tools such as creating a WMS or WCS display from a NetCDF raster (eg displaying a WMS using ncWMS, or a WCS using Thredds Data Server).

• If the user wishes to go against NetCDF conventions when exporting, they should use a -co option to specifically request this.

(2) Given (1), the driver should use the CF-1 conventions for storing needed metadata where at all possible, but where necessary add extra metadata so the file can be readable back into GDAL with no loss of information or translation error. A specific example of this is named Datum and EPSG codes, which GDAL needs to supplement CF-1.

• Again, if the user doesn't wish to supplement the files with GDAL metadata, they should specifically disable this using the appropriate -co option.

(3) Given (2), when importing from NetCDF files, the driver will first read from CF-1 conventions, then read supplementary data from GDAL-added attributes. If the two sources conflict, we should report this and consider the CF-1 encoded info as authoritative, which assumes a user has modified the data with a non-GDAL program.

Planned Changes

Change to a default south-up orientation for NetCDF data

Related tix: #4251

New -co options: ?? Should we include an option to allow force assume south-regular?

Has back-compat impact?: Yes - since affects default import and export NetCDF behaviour.

Change:

When GDAL exports to NetCDF, translate the data so that it is in "south up" orientation, i.e. invert the data from the GDAL default. Likewise upon import from CF-1, assume the data is in "south up" orientation, and translate the y values by default.

Rationale:

CF-1's default orientation is south-up, and all known current CF-1 -compliant applications expect this.

Continue saving NetCDF custom attribute tags by default, but add option not to save

Related tix: N/A
New -co options: -co WRITEGDALTAGS (yes/no, defaults to yes)
Has back-compat impact?: No

Change:

Currently, as well as following the CF-1 conventions to save projection metadata, the NetCDF driver also writes metadata in several custom attributes:

• A "spatial_ref" tag containing full OGC WKT string, including datum and EPSG codes if known
• A geotransform array, saved as a string (GeoTransform).

The WRITEGDALTAGS driver creation option would allow the user to choose not to write these tags. However it would be set to 'YES' by default.

Rationale:

We assume by default that if users export to a NetCDF file from GDAL, then they may wish to import into GDAL again and retain full information. Currently, the CF-1 convention doesn't allow saving all the projection information that GDAL can manipulate via a WKT (such as named Datums, and EPSG authorities).

However, there is certainly a use-case to export to a NetCDF file with no intention to access again via GDAL and use purely for use by CF-1 applications, in which case we don't need to save the added metadata.

Add saving of Datum reference Ellipsoid parameters in CF-1 compliant format, in addition to within GDAL's WKT

Related tix: #2893
New -co options: No, though related to the new proposed WRITEGDALTAGS option above.

Has back-compat impact?: No

Change:

When exporting to NetCDF, save CF-1 attributes that specify the reference ellipsoid used by the projection SRS's datum. Continue also saving this information in the special GDAL "spatial_ref" tag.

Note: when importing, the GDAL WKT datum information would be checked first as it fully specifies the datum. The CF-1 ellipsoid information would be usable as a fall-back.

((?? pds: Should we add anything here about special import behaviour, e.g. whether to auto-detect datum on import based on ellipsoid values?? Or should there be an option for this?))

Rationale:

CF-1 convention's only support of datums currently is to specify a reference ellipsoid using parameters such as 'semi_major_axis' and 'inverse_flattening'.

While these attributes alone do not fully specify the datum (which requires an EPSG code so control points can be referenced), they at least allow fairly accurate map projection of the created NetCDF file using tools such as ncWMS.

In the longer-term it would be beneficial to work with the CF-1 community to allow saving of full datum information in a format readable by CF-1.

When exporting to NetCDF map projections via GDAL, save projection coordinate variables. Optionally, also save lat and lon mapping variables.

Related tix: #2893
New -co options:

• WRITELONLAT=yes/no (default: yes for geographic, no for projected)
• TYPELONLAT=float/double (default: double for geographic, float for projected)

Has back-compat impact?: No

Change:

When exporting rasters to NetCDF, save Coordinate Variables for the projected SRS, as required by the CF-1 convention, normally saved as "x" and "y".

Optionally using -co options listed above, also write 2D 'lat' and 'lon' arrays, which are specified as part of the CF-1 convention for map projections - but are not actually required for the Unidata-maintained NetCDF Java API and applications built upon them (eg ncWMS, Thredds Data Server).

Rationale:

The CF-1 conventions specify these coordinate variables as necessary for grids, see ​http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.5/ch05s06.html ch5.6 and ​http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.5/apf.html App F in the document.

Tests with NetCDF CF-1 reading software indeed show that rasters with projected grids will not be correctly interpreted without the coordinate variables in the projected system (the NetCDF-Java API determines this, since it's used by ncWMS, ToolsUI and Thredds Data Server).

While the CF-1 convention examples also contain 2D "lat" and "lon" Variables containing the latitude and longitude at each data point, tests have shown that the NetCDF Java API can correctly interpret CF-1 data without these added. The rationale for this point is given at ​http://www.mail-archive.com/cf-metadata@cgd.ucar.edu/msg00759.html.

Thus we've chosen to make it optional to write these Variables, as they are redundant from GDAL's point of view (Lat and Lon are calculated on-the-fly), and they increase the size of a NetCDF file significantly. (E.g. if the actual band-data is stored as a 2D Byte array, but the lat and lon are stored as doubles, then the file-size can easily increase by 10X).

Compatibility Issues

As specified above in the sub-sections.

Test Suite

The test suite would be updated to check CF-1 compliance of exported datasets, and also that round-trip conversion from another format (eg GeoTiff) to NetCDF and back is successful.

Documentation

The GDAL driver summary page (​http://www.gdal.org/frmt_netcdf.html) would be updated with description of new behaviour and options.

Addenda

• We also intend to improve interface for writing to NetCDF 4 - (but will handle this separately, may not need an RFC)