wiki:NetCDF_Improvements

Context Navigation

Version 33 (modified by etourigny, 12 years ago) ( diff )
--

NetCDF Improvements

This page is for information regarding improvements to the GDAL NetCDF driver.

For information on the current NetCDF driver, please consult:

NetCDF - GDAL NetCDF wiki entry.
http://www.gdal.org/frmt_netcdf.html - format docs.

Topics

UNIDATA NetCDF Conventions

UNIDATA recommends the use of the CF Conventions (NetCDF Climate and Forecast (CF) Metadata Conventions) where applicable.
It is intended that this driver fully support the CF-1.5 Conventions for import and export and COARDS for import. Provisions should be taken for back-wards compatibility with different CF-1.x versions.

NetCDF Conventions at UNIDATA
CF-1.5 - NetCDF Climate and Forecast (CF) Metadata Conventions
COARDS conventions (1995 standard that CF Conventions extends and generalizes)
CF compliance checker (python script) - don't use to validate grid structure

Issues with the current implementation

NetCDF created by GDAL not CF-1.0 compliant for geographical grids (issue #2129) - fixed in trunk
netcdf driver does not read and write CF-1.0 coordinate system properly for projected CRS (issue #2893) - mostly fixed in trunk
- support for read and write of projected coordinate values - fixed
- export std_parallel variables into a single standard_parallel netcdf attribute - fixed
- add proper datum import and export support - fixed
- issues with specific projections (see wiki:NetCDF_ProjectionTestingStatus and generic report in #2893):
  - rotated_pole (#4285 and #4251)
  - Stereographic vs. Oblique_Stereographic (#4267)
  - LCC1SP - no scale_factor in CF-1 (#3324)
  - CEA : how to deal with scale_factor_at_projection_origin vs. of standard_parallel1
  - PS: how to deal with standard_parallel vs. scale_factor_at_projection_origin
add CF compliance checker to the autotests (issue #4233) - fixed
support "coordinates" attribute for projections which are not supported by GDAL/PROJ.4 (i.e rotated pole) (issues #4285 and #4251)

Metadata

GDAL and NetCDF support metadata in different ways, which should be harmonised.

Issues with the current implementation

The import process does not use the Conventions attribute and does not consider different possible Conventions (COARDS, CF-1.0, CF-1.5, etc.)
netcdf history global attribute should be updated. Does GDAL offer a history mechanism? - needs new function in gdal.h (issue #4297)

Issues that have been resolved (in trunk)

Duplication in Metadata when translating from NetCDF to NetCDF (issue #4204)
metadata from a NetCDF variable should be stored inside the GDAL Band metadata, instead of a global metadata item (issue #4211)
NetCDF variable name is lost when imported into GDAL (issue #4211)
Floating-point metadata in netcdf file is converted to GDAL metadata as string without sufficient precision (issue #4211)
GDAL version saved in GDAL global attribute
CF-version of file read could be kept in an internal variable, as well as CF-version for file creation - in trunk CF-1.5 is exported to file only

Datum issues

CF supports Datum information but does not specify how datum ids (such as those found in EPSG codes) should be stored in the netcdf file

Which datum should be set when there is no datum information at all in a netcdf file? WGS84 or none?
How to export datum information that is present in a GDAL file, in a way that it can be restored later? - current codes uses spatial_ref
Import should correctly set appropriate CRS/datum information (e.g. WGS84) from the information in the netcdf file, when sufficient information is available. - in trunk compares WKT and CF grid_mapping and checks for conflict. There are issues with auto-detecting named CRS/datum based on spheroid only.

Dimension/grid issues

The driver should handle properly a time and Z-axis in a way that they they are not lost when importing and exporting.
Add support for irregular grids (bug #1894)

I/O issues

Disabled PAM support, as it created incompatibilities and metadata duplication. Will re-enable once things are settled - #4244 and #4245
Support Create() method so that gdalwarp can be used on a netcdf file - #4221 (committed to trunk)
~~Fix access to different netcdf file types: nc (classic), nc2 (64-bit) and nc4 (NetCDF4 with HDF5 support) - #3890 and #2379~~
Add option for file type (nc,nc2,nc4) at export (#3166) -> fixed in trunk, but needs testing in Win32
Support compressed netcdf (DEFLATE with NC4 or NC4C) (#4404, #4221) -> fixed in trunk
fix the y-axis for import and export (issues #4251 and #4284) - in trunk option for bottom-up
support packed data (with add_offset and scale_factor) transparently (see bug #3797) - will not implement as no sufficient need
support for signed/unsigned Byte data - #4221 (committed to trunk)

References:

Various bugs that need fixing

support import of unequally-spaced grids
support of lon/lat grids from CF grids which are not supported
can we define dim vars before band vars, without performance hit? ncenddef??
should we set NO_FILL for performance gain? nc_set_fill(ncid, NC_NOFILL, &old_fill_mode);
implement setmetadata and setmetadataitem, tricky because other nc attributes are already written to file

Back-porting to 1.8 branch

metadata and precision fixes ( #4200, #4204, #4211, #4244 )
import of NC2, NC4, NC4C, HDF4 (#3890, #2379, #4294 and #3166)
fix GT import issue (fixed in trunk r23263)
fix import of projected srs (fixed in trunk r23263 and others)
fix CF-export of geographic and projected srs(#2129 and #2893)
add GDAL version to netcdf metadata (for detection of bottom-up c.f. r23263)

Compatibility with other software

The driver should be compatible for import and/or export (with proper visualisation) with at least the following software:

ncdump from UNIDATA netcdf
CF compliance checker
ncview
GrADS - Grid Analysis and Display System
CDO - Climate Data Operator
NCO - netCDF Operators
TDS - THREDDS Data Server (i.e. ensure its WCS and WMS capabilities work)
- ncWMS - ncWMS is the package that TDS uses for WMS presentation of NetCDF files, also includes the GODIVA web viewer for same.
others to come...

Test files

A place should be set up to store NetCDF sample files, provided they are free under the terms of http://download.osgeo.org/gdal/data/COPYING .
Large files will be available at http://download.osgeo.org/gdal/data/netcdf .
Having small (typically < or ~ 10 KB ) sample files is also good so they can be directly included in SVN under the autotest/gdrivers/data directory and thus easily used by regression tests.

Contribution

Contributors are welcome and could be identified here in 3 categories: Ideas/reporting, testing, and/or development.
Discussions can take place here or the mailing list with the title "NetCDF Improvements"
Tickets can be used to track the implementation of specific items and bugfixes.
Backward compatibility issues, in particular with respect to files produced by older GDAL versions, might require voting, as in http://trac.osgeo.org/gdal/wiki/rfc1_pmc details

Proposed changes that affect interface or backward compatibility

(Note: this linked page may be turned into an RFC if needed)

Discussion

I (etiennesky) have created this wiki entry after a discussion in the gdal-dev mailing list (Subject: discussion on improvements to the NetCDF driver and CF-1 convention). Even made a number of suggestions which I have followed (I also shamelessly copied some of his wording).
I encourage interested parties in participating in this discussion here and in the mailing list. In particular, it would be useful to collect information on contributors and obtain test files with issues and the expected outcome. Simple files can also be included in order to test for regression issues in the future.

Regards, etiennesky - Aug. 28

Open Tickets

No results

Closed Tickets

#906: NetCDF segmentation fault
#1506: gdalinfo on a netCDF file returns incorrect coordinates
#1568: gdalinfo on a netCDF file returns uninitiated coordinate information if variable/band not specified
#1625: problems reading etopo2v2 (arc grid, / GMT (netcdf) /HDF)
#1637: patch for 1.4.x : frmts/netcdf (NC_GLOBAL#{spatial_ref/GeoTransform})
#1660: add support for scale and offset in netCDF driver.
#1767: NetCDF driver confused by paths with drive letters
#1808: NetCDF Driver Improvements
#1894: NetCDF file lacks georeferencing
#1926: Corrects for the wrong header info printed by gdalinfo on GMT grids and the ReadAttributes method issue
#2000: use of uninitialized memory in netCDFDataset::SetProjection
#2129: NetCDF created by GDAL not CF-1.0 compliant for geographical grids
#2196: NetCDF driver segfaults when reading large attributes
#2379: GDAL with NetCDF4 - Adjustment to recognize netCDF4
#2540: [PATCH] Band selection in NETCDF driver
#2581: netcdf: add metadata record for original array dimension when reading from multidimensional files
#2582: five dimensional netcdf file crashes gdalinfo on windows
#2583: [PATCH] five dimensional file image access fails for some dimensions
#2584: NetCDF Global EASE-Grid 8-day Blended SSM/I and MODIS Snow Cover flipped east/west?
#2591: NetCDF Driver should read/write data in stripes instead of the whole array
#2599: accessing NetCDF band data of type char causes errors
#2654: [PATCH] Fix flipped latitude with NetCDF CF-1 files
#2657: --with-netcdf=diretory does not really test netcdf
#2846: flipped NetCDF driver output
#2893: netcdf driver does not read and write CF-1.0 coordinate system properly for projected CRS
#3154: Configure snippet is not truly able to manage both netcdf3 and netcdf4
#3166: can't make large/64-bit netCDF file
#3184: unable to compile 1.6.2 w/ netcdf, hdf
#3324: Add Lambert Conformal Conic single standard parallel support to netCDF driver
#3425: Add new projections from cf-1.4 to netcdf driver
#3520: Lambert_Conformal_Conic_1SP incorrectly read as Lambert_Conformal_Conic_2SP by netcdf driver
#3562: netCDF for western hemisphere
#3563: netCDF for western hemisphere
#3715: GDAL's NetCDF driver doesn't seem to read the spatial reference system as documented
#3797: netCDF driver ignores the "scale_factor"
#3863: Error while reprojecting with gdalwarp an ECMWF netCDF
#3890: netCDF file reading error
#3957: Configuring gdal 1.8.0 with hdf and netcdf at the same time fails
#3976: Problem creating netcdf files with numerous bands
#4047: gdal_translate segfault when translating ERS to NetCDF.
#4145: upside flipped NetCDF
#4200: [PATCH] gdal_translate GeoTIFF to NetCDF results in coordinates shifting
#4204: Duplication in Metadata when translating from NetCDF to NetCDF
#4221: Support Create() method for NetCDF driver so that gdalwarp can be used on a netcdf file
#4233: netcdf: add CF compliance checker to the autotests
#4245: PAM stores duplicate metadata even if the format and driver support
#4251: netcdf driver does not support rotated_pole projection
#4284: Change netcdf driver to read and write in the bottom-up order by default
#4285: OGR does not support PROJ.4 +proj=ob_tran , rotated_pole is not supported by the netcdf driver
#4294: changes to the netcdf driver export and import
#4297: add support for "history" metadataitem which keeps track of all GDAL command line operations
#4402: Issue with netcdf driver and srs units
#4404: gdal_translate GeoTIFF to NetCDF with NC4 DEFLATE compression is taking forever
#4405: Error reading netCDF dataset
#4408: GDAL configure doesn't pick up NetCDF4 in latest Ubuntu 11.10
#4424: configure script does not support static netcdf build
#4432: gdal_rasterize produces error when burning vector dataset into netCDF dataset directly with -a_nodata nan
#4437: lack of closing read-only dataset dataset or calling GDALDestroyDriverManager() on NC4C (netCDF-4 classic model) file producing HDF5-DIAG error messages
#4484: gdalwarp on large netcdf datasets sets data from first input dataset to <nodata> and generates error if -dstnodata option is used with netcdf-4 file
#4512: netcdf files with longitude between 180 and 360 degrees are incorrectly projected
#4513: netcdf driver does not support irregular grids / GDAL GEOLOCATION arrays
#4514: netcdf driver does not keep original latitude values for Gaussian grids
#4554: gdalwarp does not properly close datasets in error conditions, can output errors in shared library unloading
#4709: NCDFAddGDALHistory() uses free'd memory
#4769: NetCDF driver assumes 'x' 'y' to be names of dimension variables
#4886: NetCDF driver writes incorrect GeoTranform in case of WRITE_BOTTOMUP=YES
#4890: PATCH: DODS driver can now read 3-D datasets, metadata, projection, remote NetCDF files
#4974: onestep vs twosteps warping to netcdf
#5084: duplicate nodata metadata in netcdf files copied with gdal_translate
#5114: netcdf driver incorrectly calculates geotransform
#5118: NetCDF driver doesn't recognize grids as uniform.
#5291: netcdf driver does not detect .grd files in netcdf-4 format / netcdf driver does not support netcdf-4 files with default chunking and bottom-up format / hdf5 driver crashes when H5Sget_simple_extent_ndims() returns negative value
#5389: NetCDF: gdal_translate unable to translate EASE-GRID file properly
#5802: gdal HDF5 NetCDF plugin bug
#5873: netcdf driver does not detect "top-down" order
#5935: GTiff to netcdf conversion fails
#5950: [PATCH] netCDF: optimize IReadBlock() and CheckData() handling of partial blocks in the x axis
#5962: gdalinfo segmentation fault reading NetCDF + CF grid
#6175: netCDF classic reports negative nodata value for unsigned byte > 127
#6227: Buffer overflow potential in netcdf driver
#6231: 1 byte heap write overflow in NCDFTokenizeArray()
#6244: netCDF: lack of validation that gridmapping:GeoTransform has 6 values
#6245: NCDFIsGDALVersionGTE(): lack of input validation
#6337: gdalinfo and gdal_translate seg faulting for unsigned short variables in netcdf file
#6551: allow vector read of NetCDF NASA ocean color L3Bin files
#6613: gdal_translate returning PROJCS "unnamed" for lambert_cylindrical_equal_area
#6645: segfault in NCDFGet1DVar
#6913: SM2RAIN-ASCAT global precipitation netcdf file cannot be read
#6942: Using H5S_MAX_RANK rather than hard coded 32?
#7172: NetCDF crashes ogrinfo and QGIS.

Note: See TracWiki for help on using the wiki.

Download in other formats:

Plain Text