Opened 7 years ago

Closed 7 years ago

Last modified 7 years ago

#6856 closed defect (invalid)

fix support for non-classic netCDF4

Reported by: jacobst Owned by: warmerdam
Priority: normal Milestone:
Component: default Version: 2.1.1
Severity: normal Keywords:
Cc:

Description

While netCDF driver has seen significant improvements, its documented to support non-classic netCDF4, but faces problems with unsigned integer variables in the netCDF (one of the differences between classic and non-classic models of netCDF4).

Tried with gdal v2.1.1, shipped with QGIS Las Palmas: When gdal_translate is performed from input netCDF4 to output VRT, unsigned data types are reported as 'unsupported' and changed to Float32, which can only mess up the data read.

As work-around, the compagnon HDF5 driver can be used to actually access the unsigned integer data.

Proof of concept:

  • take netCDF4 with unsigned byte or unsigned short data layer
  • gdal_translate to VRT format, which issues a warning on data type
  • edit the VRT, then change 'Float32' back to the correct data type
  • and change the VRT's source file reference from NETCDF:some_file:some_layer into HDF5:some_file://some_layer

The data can then be read through the VRT file.

Attachments (2)

forGDALtrac.zip (210.8 KB ) - added by jacobst 7 years ago.
set of gdalinfo and gdal_translate'd VRT files
gdalinfo_format_nc.txt (2.7 KB ) - added by jacobst 7 years ago.
output from gdalinfo --format netcdf

Download all attachments as: .zip

Change History (14)

comment:1 by Even Rouault, 7 years ago

Can you attach a file or point to a file that demonstrates the issue ? From a quick skim and testing of the code, uint32 should be handled.

comment:2 by jacobst, 7 years ago

Sample files I have are much too big to be attached to the ticket (1MB size limit). If you email me, than I can send it to you via eg dropbox link.

by jacobst, 7 years ago

Attachment: forGDALtrac.zip added

set of gdalinfo and gdal_translate'd VRT files

comment:3 by jacobst, 7 years ago

I've just attached several small text files:

  • outputs of h5dump -p -A, ncdump -c (to illustrate the structure of the file)
  • outputs of gdalinfo, using both HDF5 and NetCDF drivers, at both level of the entire file and at level of the NDVI subdataset (or band) in the file
  • VRT files produced with HDF5 and NETCDF drivers => note the difference in data type for the band that is described (Byte vs Float32)

Warning text that is displayed: Warning 1: Unsupported netCDF datatype (7), treat as Float32.

The priority is to fix the data read. But concerning the warning text itself, I can further note that:

comment:4 by Even Rouault, 7 years ago

You can email me the dropbox link at even.rouault at spatialys.com

comment:5 by jacobst, 7 years ago

dropbox link was just sent

comment:6 by Even Rouault, 7 years ago

Ah, can you send instead to even.rouault at gmail.com because of Dropbox identification...

comment:7 by jacobst, 7 years ago

done

comment:8 by Even Rouault, 7 years ago

It is working fine for me on Ubuntu 16.04 with libnetcdf 4.4.0-2. Are you sure your GDAL build is made against libnetcdf V4 , and not an older version ? This really looks like if that was the case.

#ifdef NETCDF_HAS_NC4
    /* NC_UBYTE (unsigned byte) is only available for NC4 */
    else if( nc_datatype == NC_UBYTE )
        eDataType = GDT_Byte;
    else if( nc_datatype == NC_USHORT )
        eDataType = GDT_UInt16;
    else if( nc_datatype == NC_UINT )
        eDataType = GDT_UInt32;
#endif
    else
    {
        if( nBand == 1 )
            CPLError( CE_Warning, CPLE_AppDefined,
                      "Unsupported netCDF datatype (%d), treat as Float32.",
                      (int) nc_datatype );
        eDataType = GDT_Float32;
        nc_datatype = NC_FLOAT;
    }

comment:9 by jacobst, 7 years ago

How would I found out this version for a pre-compiled NETCDF driver that was shipped with QGIS Las Palmas?

ncdump reports libnetcdf 4.3.0, but it is from a different install, so I'm not sure QGIS' copy of GDAL would use the same lib under its hood.

comment:10 by Even Rouault, 7 years ago

Well, this depends on the operating system: Linux, Windows (OSGeo4W ?)... ?

If you can access to a shell with the GDAL version used by your QGIS, you can type

$ gdalinfo --format netcdf

If you see NC4 mentionned in the output, like in

<CreationOptionList>
  <Option name="FORMAT" type="string-select" default="NC">
    <Value>NC</Value>
    <Value>NC2</Value>
    <Value>NC4</Value>
    <Value>NC4C</Value>
  </Option>

then this should work like in my setup. But from what you reported and the source code, I'm pretty sure the issue is that the GDAL build you use doesn't enable NC4.

Last edited 7 years ago by Even Rouault (previous) (diff)

by jacobst, 7 years ago

Attachment: gdalinfo_format_nc.txt added

output from gdalinfo --format netcdf

comment:11 by Even Rouault, 7 years ago

Resolution: invalid
Status: newclosed

OK the output of gdalinfo confirms that your GDAL build is not netCDF v4 enabled. Something to report to the binary distribution that provides GDAL to you

Closing as not a GDAL bug

comment:12 by jacobst, 7 years ago

I've attached the output of the gdalinfo --format netcdf command. You seem to be on the right track. It lists just NC and NC2, not NC4/NC4C formats.

OS is Windows 7, 64bit with Service Pack 1 GDAL was included in stand-alone QGIS installation file QGIS-OSGeo4W-2.18.0-1-Setup-x86_64.exe downloaded from qgis.org.

Note: See TracTickets for help on using tickets.