= NetCDF Improvements = This page is for information regarding improvements to the GDAL NetCDF driver. For information on the current NetCDF driver, please consult: * [wiki:NetCDF] - GDAL NetCDF wiki entry. * http://www.gdal.org/frmt_netcdf.html - format docs. = Topics = == UNIDATA NetCDF Conventions == UNIDATA recommends the use of the CF Conventions (NetCDF Climate and Forecast (CF) Metadata Conventions) where applicable.[[BR]] It is intended that this driver fully support the CF-1.5 Conventions for import and export and COARDS for import. Provisions should be taken for back-wards compatibility with different CF-1.x versions. * [http://www.unidata.ucar.edu/software/netcdf/docs/conventions.html NetCDF Conventions at UNIDATA] * [http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.5/cf-conventions.html CF-1.5 - NetCDF Climate and Forecast (CF) Metadata Conventions] * [http://ferret.wrc.noaa.gov/noaa_coop/coop_cdf_profile.html COARDS conventions (1995 standard that CF Conventions extends and generalizes)] * [http://cf-pcmdi.llnl.gov/conformance/compliance-checker/ CF compliance checker] [http://pypi.python.org/pypi/cfchecker/2.0.2 (python script)] - don't use to validate grid structure === Issues with the current implementation === * ~~NetCDF created by GDAL not CF-1.0 compliant for geographical grids (issue #2129)~~ * netcdf driver does not read and write CF-1.0 coordinate system properly for projected CRS (issue #2893) - mostly fixed in trunk and 1.9 * ~~support for read and write of projected coordinate values~~ * ~~export std_parallel variables into a single standard_parallel netcdf attribute~~ * ~~add proper datum import and export support~~ * issues with specific projections (see wiki:NetCDF_ProjectionTestingStatus and generic report in #2893): * rotated_pole (#4285 and #4251) * ~~Stereographic vs. Oblique_Stereographic (#4267)~~ * ~~LCC1SP - no scale_factor in CF-1 (#3324)~~ * CEA : how to deal with scale_factor_at_projection_origin vs. of standard_parallel1 * ~~PS: how to deal with standard_parallel vs. scale_factor_at_projection_origin~~ * ~~add CF compliance checker to the autotests (issue #4233)~~ * support "coordinates" attribute (and associated curvilinear grid) for projections which are not supported by GDAL/PROJ.4 (i.e rotated pole) (issues #4285 and #4251) == Metadata == GDAL and NetCDF support metadata in different ways, which should be harmonised. === Issues with the current implementation === * The import process does not use the Conventions attribute and does not consider different possible Conventions (COARDS, CF-1.0, CF-1.5, etc.) * ~~netcdf history global attribute should be updated. no mecanism for saving commands in gdal, due to security issues (issue #4297)~~ * implement setmetadata and setmetadataitem, tricky because other nc attributes are already written to file === Issues that have been resolved (in trunk and 1.9) === * Duplication in Metadata when translating from NetCDF to NetCDF (issue #4204) * metadata from a NetCDF variable should be stored inside the GDAL Band metadata, instead of a global metadata item (issue #4211) * NetCDF variable name is lost when imported into GDAL (issue #4211) * Floating-point metadata in netcdf file is converted to GDAL metadata as string without sufficient precision (issue #4211) * GDAL version saved in GDAL global attribute * CF-version of file read could be kept in an internal variable, as well as CF-version for file creation - in trunk and 1.9 CF-1.5 is exported to file only == Datum issues == CF supports Datum information but does not specify how datum ids (such as those found in EPSG codes) should be stored in the netcdf file * Which datum should be set when there is no datum information at all in a netcdf file? WGS84 or none? * How to export datum information that is present in a GDAL file, in a way that it can be restored later? - current codes uses spatial_ref * Import should correctly set appropriate CRS/datum information (e.g. WGS84) from the information in the netcdf file, when sufficient information is available. - in trunk and 1.9 compares WKT and CF grid_mapping and checks for conflict. There are issues with auto-detecting named CRS/datum based on spheroid only. == Dimension/grid issues == * The driver should handle properly a time and Z-axis in a way that they they are not lost when importing and exporting (bug #2581) * Add support for irregular grids (bug #1894) * Check the axis attribute on the dims to determine the correct order (instead of assuming the order of x, y, z, t) * Irregular grid testing: use a fraction threshold, not an absolute amount (0.1 degrees at present). == I/O issues == * ~~Disabled PAM support, as it created incompatibilities and metadata duplication. Will re-enable once things are settled~~ PAM has been re-implemented to store raster statistics only - #4244 and #4245 * ~~Support Create() method so that gdalwarp can be used on a netcdf file - #4221 (committed to trunk)~~ * ~~Fix access to different netcdf file types: nc (classic), nc2 (64-bit) and nc4 (NetCDF4 with HDF5 support) - #3890 and #2379~~ * ~~Add option for file type (nc,nc2,nc4) at export (#3166)~~ * ~~Support compressed netcdf (DEFLATE with NC4 or NC4C) (#4404, #4221)~~ * ~~fix the y-axis for import and export (issues #4251 and #4284)~~ * ~~support packed data (with add_offset and scale_factor) transparently (see bug #3797) - will not implement~~ * ~~support for signed/unsigned Byte data - #4221~~ * ~~Ignore coordinate and bounds variables (CF sections 5.2, 5.6 and 7.1) as raster bands, but expose them as subdatasets - this allows to open files with projected SRS (or dimension bounds) directly, without specifying the variable as a subdataset (added in r23977)~~ References: * http://www.unidata.ucar.edu/software/netcdf/workshops/2010/bestpractices/Packing.html * http://www.unidata.ucar.edu/software/netcdf/docs/BestPractices.html#Packed%20Data%20Values == Various bugs / wishlist / questions == * can we define dim vars before band vars, without performance hit? nc__enddef?? * should we set NO_FILL for performance gain? nc_set_fill(ncid, NC_NOFILL, &old_fill_mode); * remove stdp1 from lcc-1sp * fix metadata access, unique way (fetchcopy parm, NCDFGetAttr, getmetadata, CSLFetchNameValue(poDS->papszMetadata, szTemp); ) * fix warnings (???) * add NCDF_DEBUG option (and supporting function) to supress most debugging output unless specified == Back-porting to 1.8 branch == * metadata and precision fixes ( #4200, #4204, #4211, #4244 ) * import of NC2, NC4, NC4C, HDF4 (#3890, #2379, #4294 and #3166) * fix GT import issue (fixed in trunk r23263) * fix import of projected srs (fixed in trunk r23263 and others) * fix CF-export of geographic and projected srs(#2129 and #2893) * add GDAL version to netcdf metadata (for detection of bottom-up c.f. r23263) == Compatibility with other software == The driver should be compatible for import and/or export (with proper visualisation) with at least the following software: * ncdump from UNIDATA netcdf * [http://cf-pcmdi.llnl.gov/conformance/compliance-checker/ CF compliance checker] * [http://meteora.ucsd.edu/~pierce/ncview_home_page.html ncview] * [http://www.iges.org/grads GrADS] - Grid Analysis and Display System * [https://code.zmaw.de/projects/cdo CDO] - Climate Data Operator * [http://nco.sourceforge.net NCO] - netCDF Operators * [http://www.unidata.ucar.edu/software/tds/ TDS] - THREDDS Data Server (i.e. ensure its WCS and WMS capabilities work) * [http://www.resc.rdg.ac.uk/trac/ncWMS/ ncWMS] - ncWMS is the package that TDS uses for WMS presentation of NetCDF files, also includes the GODIVA web viewer for same. * others to come... == Test files == A place should be set up to store NetCDF sample files, provided they are free under the terms of http://download.osgeo.org/gdal/data/COPYING . [[BR]] Large files will be available at http://download.osgeo.org/gdal/data/netcdf .[[BR]] Having small (typically < or ~ 10 KB ) sample files is also good so they can be directly included in SVN under the autotest/gdrivers/data directory and thus easily used by regression tests. == Contribution == * Contributors are welcome and could be identified here in 3 categories: Ideas/reporting, testing, and/or development. * Discussions can take place here or the mailing list with the title "NetCDF Improvements" * Tickets can be used to track the implementation of specific items and bugfixes. * Backward compatibility issues, in particular with respect to files produced by older GDAL versions, might require voting, as in http://trac.osgeo.org/gdal/wiki/rfc1_pmc details = Proposed changes that affect interface or backward compatibility = (Note: this linked page may be turned into an RFC if needed) * wiki:NetCDF_exportUpdateChanges * wiki:NetCDF_ProjectionTestingStatus = Discussion = I (etiennesky) have created this wiki entry after a discussion in the gdal-dev mailing list (Subject: discussion on improvements to the NetCDF driver and CF-1 convention). Even made a number of suggestions which I have followed (I also shamelessly copied some of his wording). [[BR]] I encourage interested parties in participating in this discussion here and in the mailing list. In particular, it would be useful to collect information on contributors and obtain test files with issues and the expected outcome. Simple files can also be included in order to test for regression issues in the future. Regards, etiennesky - Aug. 28 [[BR]] = Open Tickets = [[TicketQuery(status=new|assigned|reopened&keywords~=netcdf|netCDF|NETCDF|NetCDF&order=priority)]] [[BR]] = Closed Tickets = [[TicketQuery(status=closed&keywords~=netcdf|netCDF|NETCDF|NetCDF&order=priority)]]