Opened 6 years ago

Closed 5 years ago

#4211 closed defect (fixed)

[PATCH] various fixes to NetCDF metadata import/export

Reported by: etiennesky Owned by: warmerdam
Priority: normal Milestone:
Component: GDAL_Raster Version: svn-trunk
Severity: normal Keywords:
Cc:

Description

The following issues have been identified with NetCDF metadata import/export. A patch has been created and submitted along with issue #2129 (patch-meta2.txt).

  • Duplication in Metadata when translating from NetCDF to NetCDF (issue #4204)
  • metadata from a NetCDF variable should be stored inside the GDAL Band metadata, instead of a global metadata item
  • NetCDF variable name is lost when imported into GDAL
  • Floating-point metadata in netcdf file is converted to GDAL metadata as string without sufficient precision (similar to #4200)

Attachments (4)

patch-meta2.txt (9.1 KB) - added by etiennesky 6 years ago.
patch for metatada fixes
era40-t2.nc (22.1 KB) - added by etourigny 5 years ago.
test1.nc (22.4 KB) - added by etourigny 5 years ago.
test1-cp3.nc (23.4 KB) - added by etourigny 5 years ago.

Download all attachments as: .zip

Change History (10)

Changed 6 years ago by etiennesky

Attachment: patch-meta2.txt added

patch for metatada fixes

comment:1 Changed 6 years ago by etourigny

Resolution: fixed
Status: newclosed

Fixed in trunk (r23094).

Metadata items are not copied by default, except those that are created elsewhere in GDAL, global netcdf attributes, time, lev and depth (until proper support for those dimensions).

Also added a GDAL global attribute, which outputs the same as gdalinfo --version.

comment:2 Changed 5 years ago by etourigny

Discovered 2 new issues with metadata handling:

1) All metadata is saved a NC_CHAR, but some data (like add_offset) needs to be in numerical format (NC_DOUBLE or NC_INT). Added a test to see if each metadata item represents an int, float or double, and save accordingly in the file.

2) add-offset and scale_factor are always exported, regardless if the variable has been unscaled or not. Modified the export to only write these variables if GetOffset?() and GetScale?() are different from the default values of (0,1).

Attaching a file with scale_factor and add_offset.

Fixed in trunk (r23241 and r23242).

Changed 5 years ago by etourigny

Attachment: era40-t2.nc added

comment:3 Changed 5 years ago by Even Rouault

Resolution: fixed
Status: closedreopened

Etienne,

r23241 breaks compilation with MSVC. See http://vbkto.dyndns.org:1280/sdk/build-output/vc7-20111016-4-10-57-90-vc7-dev.txt

strtof is C99, which MSVC doesn't like. Perhaps you could just use strtod, and then compare (double)(float)dfMetaValue with dfMetaValue with CPLIsEqual to determine if it is a float or a double ?

comment:4 Changed 5 years ago by etourigny

The proposed method does not work, because casting float to double add extra junk after the 7th significant digit, therefore CPLIsEqual() always returns false. What works well is to cast it to float, write it back to a char* and compare with the original metadata.

                        fMetaValue = (float)dfMetaValue; 
                        sprintf( szTemp,"%.7g",fMetaValue); 
                        if ( EQUAL(szTemp, szMetaValue ) )

In the process I discovered that floats should be written to char* with format "%.7g" and doubles should use "%.15g".

For floats, this means not always storing 7 digits (e.g. 1.004 would be saved as "1.00400")

For doubles, saving only 15 instead of 16, because netcdf data only uses 15 digits. Had to modify the autotest netcdf_8 because of this change. In the associated test file 'data/cf_aea2sp_invf.nc' std_parallel is saved in the netcdf as '29.8333333333333' (15 digits), but gdal would save it as '29.83333333333334' (16 digits) which is inconsistent. This also inconsistencies in the geotransform. Note that this fix does not cause a regression with bug #4200.

comment:5 Changed 5 years ago by etourigny

Fixed in trunk (r23243).

Attaching a test file (test1.nc) and correct output from gdal_translate (test1-cp3.nc) with various data types.

The following data

		t2m:tmpd1 = 1.0004f ;
		t2m:tmpd2 = 1.00080888444444 ;
		t2m:tmpd3 = 1.0008084543 ;
		t2m:tmpf = 1.000809f ;
		t2m:tmpl = 1 ;

now gets converted (with gdal_translate) to

		t2m:tmpd1 = 1.0004f ;
		t2m:tmpd2 = 1.00080888444444 ;
		t2m:tmpd3 = 1.0008084543 ;
		t2m:tmpf = 1.000809f ;
		t2m:tmpl = 1 ;

Note that tmpd1 gets converted to a float (instead of double) which is ok, because we lost no precision. tmpd2 and tmpd3 are saved as double, tmpf is saved as float, and tmp as int

before this fix (r23094) the values would all be saved as NC_CHAR (char*):

		t2m:tmpd1 = "1.0004" ;
		t2m:tmpd2 = "1.000808884444444" ;
		t2m:tmpd3 = "1.0008084543" ;
		t2m:tmpf = "1.000809" ;
		t2m:tmpl = "1" ;

Changed 5 years ago by etourigny

Attachment: test1.nc added

Changed 5 years ago by etourigny

Attachment: test1-cp3.nc added

comment:6 Changed 5 years ago by etourigny

Resolution: fixed
Status: reopenedclosed

Closing this bug as it was fixed in trunk (r23243).

Note: See TracTickets for help on using tickets.