wiki:ADAGUC

ADAGUC netCDF

The ADAGUC netCDF driver reads and writes files which comply with the ADAGUC Data Products Standard ADAGUC website (Atmospheric Data Access for the Geospatial User Community). ADAGUC files are created by using the NetCDF4 API. The NetCDF4 API has the ability to store files in the HDF5 file format while remaining backwards compatible with previous versions of netCDF. The ADAGUC driver supports the creation of NetCDF3 and NetCDF4 files. In NetCDF3 mode the driver writes NetCDF files, in NetCDF4 mode the driver writes HDF5 files. ADAGUC netCDF files follow the Climate and Forecast (CF) 1.2 and ISO:19115 metadata conventions. Besides these conventions the format provides space for specific product metadata.

The file structure

The ADAGUC internal file structure contains variables which can be subdivided in three types: variables to store the metadata, variables to store the dimensions and variables to store the data. A schematic overview of the ADAGUC file structure for the different types is given in the file structure figure (below on the left). The three types of variables (metadata, dimension and data) are indicated with A, B and C respectively.

(A) Metadata - Metadata is stored in attributes which are assigned to a variable. The variables ‘product’, ‘iso_dataset’ and ‘projection’ are used to store the metadata.

(B) Dimensions - Dimensions are used as coordinate variables and time intervals. They provide information about geolocation and time to the data variables. The dimensions follow the NetCDF Climate and Forecast (CF) Metadata Conventions. Dimensions are lists containing coordinates or time intervals. The dimension scale figure shows geographic raster data with the dimensions time, lat and lon. Its size is, in this example, 4x6x12. This means that there are 4 indexes for time, 6 for latitude and 12 for longitude. According to the CF conventions the dimensions should have the order time, lat, lon. In case the raster data uses a geographic coordinate system, the required dimensions are time, lat and lon which represent the center of the pixel. When the raster data is projected the dimensions x and y are included which represent the projected coordinates. The dimensions lat and lon become a function of y and x (see the right part of the file structure), providing the latitude and longitude at the location of y and x, respectively. In this case y and x contain the projected coordinates, while lat and lon represent the geographic latitudes and longitudes in degrees at location y, x. For detailed information see the NetCDF Climate and Forecast (CF) Metadata Conventions.

(C) Data - This part contains the data in the file structure. The data is stored in variables which are functions of the dimensions. For geographic raster data the variable is a function of time, lat and lon. For projected raster data the variable is a function of time, y and x, as can be seen in the illustration of the file structure. The ADAGUC format currently supports X, Y and time dimensions, stored as dimensions in the file according the CF Conventions. The driver supports two ways of time sub-setting. The first is by making selections in time by choosing band numbers, the second is by providing the time in the file name. By using gdalinfo on the ADAGUC file the available band numbers and dimensions are listed. With this information a subset can be read with for example:

gdalinfo ADAGUC:thefile.nc:variable:time=20

Left: filestructure; Right: netcdf dimensions


GDAL Creation options

Creation options can be used to provide additional metadata and set the behavior of the driver. Metadata attributes set as a creation option override already existing metadata attributes.

METANCML – It is possible to provide additional metadata which is stored in an NcML XML file. The attributes from the groups product, iso_dataset, projection and custom are copied. When the name of the dataset is specified in the product::variables attribute, specific dataset attributes can be placed under the group with the name of the dataset. A sample file is inluded in the sample ADAGUC zip file (see section sample ADAGUC file).

FORCENC3 – When set to true, the NetCDF4 driver writes in NetCDF3 mode. When unspecified the value is FALSE, in this case the driver writes in NetCDF4 mode.

VALSTART and VALSTOP – Overwrites the product validity_start and product validity_stop attributes in the product group. For example: –co “VALSTART=20080921T101900”

DATASETNAME - Specifies the name of the dataset

LONGNAME, STDNAME and UNITS – Sets the long_name, standard_name and units attributes of the dataset.

NAN – Sets the no data value of the dataset. The _FillValue attribute of the dataset is filled with the NAN number.


Building the driver

Download the required libraries
The ADAGUC netCDF driver requires at least the netCDF-4.0, HDF-1.8.1 and UDUNITS-1 packages to be installed.

Configuring the libraries

  • HDF5 library:
    ./configure --prefix=<installdir> --with-default-api-version=v16
  • netCDF4 library:
    ./configure --prefix=<installdir> --with-hdf5=<installdir> --enable-netcdf-4 --enable-cxx-4 --enable-shared
  • UDUNITS library:
    Un SUSE I had to export several environment variables to make it work:
    export FC=g77
    export CPPFLAGS=-Df2cFortran
    export CC=gcc
    export CXX=g++
    export LD_MATH=-lm
    ./configure --prefix=<installdir>

Installing the GDAL ADAGUC driver

The driver consists of two files, adagucdataset.h and adagucdataset.cpp. These should be copied to the gdal/frmts/netcdf/ directory, so they reside besides the netCDF and GMT driver.

  • 1. Copy adagucdataset.h and adagucdataset.cpp to the ./<gdaldir>/frmts/netcdf/ directory
  • 2. Update the GNUmakefile and makefile.vc in the ./<gdaldir>/frmts/netCDF/ directory by adding the object with name: adagucdataset.o to the already existing objects.
  • 3. Add the registration entry point declaration GDALRegister_ADAGUC() to gdal/gcore/gdal_frmts.h, at the location above the already existing netCDF driver:

In gdal/gcore/gdal_frmts.h:

…
void CPL_DLL GDALRegister_GMT(void);
void CPL_DLL GDALRegister_ADAGUC(void);
void CPL_DLL GDALRegister_netCDF(void);
…
  • 4. Add a call to the registration function to frmts/gdalallregister.c, in the already existing netcdf ifdef:

In frmts/gdalallregister.c:

…
#ifdef FRMT_netcdf
GDALRegister_GMT();
GDALRegister_ADAGUC();
GDALRegister_netCDF();
#endif
…

In step 3 and 4 the ADAGUC register entries should be put above the netCDF register entry, to make sure that ADAGUC files are not opened by the netCDF driver. The ADAGUC driver will only open ADAGUC files, by checking the existence of the iso_dataset and product variables.

Configuring GDAL
To make sure that the libraries are found during the compilation process you can use ./configure LIBS="-lhdf5 -lhdf5_hl -ludunits $LIBS".
Use --with-netcdf=<path to install tree> to build GDAL with the NetCDF library.


Sample ADAGUC files

[This zip file contains a folder with sample ADAGUC files. The folder contains two different datasets. The first contains Sciamachy level 3 tropospheric NO2 measurements obtained from http://www.temis.nl. The second file contains precipitation radar data from the Royal Dutch Meteorological Institute (KNMI, http://www.knmi.nl). The radar dataset has been resampled to 256x256 pixels to make it suitable for this example. The radar dataset contains 24 Radar precipitation images with 5 minute interval ranging from 2008-07-20 00:00:00 till 2008-07-201 01:55:00. This radar dataset is included purely to demonstrate the use of the time dimension. The file is created from a source ADAGUC NetCDF4 file (not included). When translating from an ADAGUC file all time dimensions with corresponding units are copied. It is also possible to subset time by using the select band option -b. Multiple bands can be selected. A single image can also be opened by using ADAGUC:filename.nc:varname.nc:dimname=value, e.g. ADAGUC:RADAR_EXP.NC:NL25_PCP:time=45 The first NO2 dataset is fully ADAGUC compliant and contains metadata obtained from the NCML file.


gdal commands

Create NetCDF4/HDF5 file:

    gdal_translate -of ADAGUC\
     -co "METANCML=SCIA__TEST_R___TMTNO2__L3.xml"\
     -co "VALSTART=20080601T00:00:00.00000"\
     -co "VALSTOP=20080701T00:00:00.00000"\
     SCIA__TEST_R___TMTNO2__L3__20080601T000000_20080701T000000.asc\
     SCIA__TEST_R___TMTNO2__L3__20080601T000000_20080701T000000.h5

Create NetCDF3 file:

    gdal_translate -of ADAGUC\
     -co "METANCML=SCIA__TEST_R___TMTNO2__L3.xml"\
     -co "VALSTART=20080601T00:00:00.00000"\
     -co "VALSTOP=20080701T00:00:00.00000"\
     -co "FORCENC3=TRUE"\
     SCIA__TEST_R___TMTNO2__L3__20080601T000000_20080701T000000.asc\
     SCIA__TEST_R___TMTNO2__L3__20080601T000000_20080701T000000.h5

Create Resampled Precipitation Radar NetCDF3 file with time dimension:

     gdal_translate -of ADAGUC\
     -co "FORCENC3=TRUE" -outsize 256 256 -co "NAN=255"\
     RADAR_EXP.h5\
     RAD___TEST_R_C_NL25PCP_L___20080720T000000_200807201T015500_0001_resampledto256x256.nc

Copy a subset from the radar file to a GeoTiff? file:

     gdal_translate -of GTiff ADAGUC:inputfile.nc:NL25_PCP:time=45 outputfile.tif

View ADAGUC files

The driver supports writing in NetCDF3 and NetCDF4 mode. In case of NetCDF3, the files can be viewed with:

In case of NetCDF3, HDF5 or NetCDF4, the files can be viewed with:

The radar dataset can be animated by Unidata's IDV.


TODO

This is the first release of the driver. Therefore it possible that there are glitches using the driver. The driver has been tested with GDAL 1.5.2 on SuSe? linux.
There are still things left to do:

  • Support for scale and offset factors has not yet been implemented
  • Bug testing!

Thanks to

Thanks to Denis Nadeau and Frank Warmerdam. This driver has been created on basis of the original netCDF driver from Denis Nadeau and Frank Warmerdam


Driver Updates - Currently Version 0.2 is available

At 7 October 2008, version 0.2 of the GDAL ADAGUC Raster driver has been released. Several bugs have been removed and functionality has been added:

  • Support for metadata attribute types has been added:
NCML type definitionNetCDF TypeGDAL metadata prefix
StringNetCDF Text[C]
shortNC_SHORT[S]
intNC_INT[I]
floatNC_FLOAT[F]
doubleNC_DOUBLE[D]
  • When translating between ADAGUC NetCDF4 files the metadata attribute types are kept. The GDAL API is used to pass metadata in the form of Strings between drivers. Inside the ADAGUC NetCDF driver the metadata has the form of "NO2#[C]long_name=Nitrogen Dioxide". In this case the attribute 'long_name' with value 'Nitrogen Dioxide' is put as string to variable 'NO2'. '[C]' indicates that the metadata type is a String.
  • The attribute 'input_products' from the product variable is now correctly updated
  • The attribute 'history' from the product variable now lists dates at which the history was updated
  • Datasets with GDAL datatypes Byte, Int16, Int32, Float32 and Float64 are now supported
  • Float64 datasets with a Time dimension are now correctly copied

Installation instructions for UBUNTU and GDAL 1.10 - 2013-07-26

Here is a short guide for installing the GDAL ADAGUC driver on Ubuntu.

sudo apt-get install libcurl4-openssl-dev libcairo2-dev libxml2-dev libgd2-xpm-dev libproj-dev 
sudo apt-get install libudunits2-dev udunits-bin

# Install latest hdf,netcdf and gdal libraries
# Source is put into /data/software, binaries are put into /data/build
# Note that slightly older versions of HDF5 and NetCDF4 can be installed with the packagemanager as well.

export CPPFLAGS="-I/data/build/include/"
export LDFLAGS="-L/data/build/lib/"
export LD_LIBRARY_PATH="/data/build/lib/:$LD_LIBRARY_PATH"
export PATH="/data/build/bin/:$PATH"

# You can add these exports to your ~/.bashrc file


### INSTALL HDF5 from source ###
cd /data/software
wget http://www.hdfgroup.org/ftp/HDF5/current/src/hdf5-1.8.11.tar.gz
tar -xzvf hdf5-1.8.11.tar.gz 
cd hdf5-1.8.11/
./configure --prefix=/data/build
make
make install


### INSTALL NetCDF4 from source ###
cd /data/software
wget http://www.unidata.ucar.edu/downloads/netcdf/ftp/netcdf-4.3.0.tar.gz
tar -xzvf netcdf-4.3.0.tar.gz 
cd netcdf-4.3.0/
./configure --prefix=/data/build --enable-netcdf-4
make
make install


### INSTALL GDAL from source with the ADAGUC driver###
cd /data/software
wget http://download.osgeo.org/gdal/1.10.0/gdal-1.10.0.tar.gz
tar -xzvf gdal-1.10.0.tar.gz 

# Install GDAL ADAGUC driver (optional)
cd gdal-1.10.0/frmts/netcdf

wget http://trac.osgeo.org/gdal/raw-attachment/wiki/ADAGUC/GDAL_ADAGUC_source_v0.3.tar.gz
tar -xzvf GDAL_ADAGUC_source_v0.3.tar.gz

# Move source files to this directory
mv GDAL_ADAGUC_source_v0.3/* .

# Adjust GNUMakefile with e.g. vi and add  adagucdataset.o to the OBJ list

# Register the ADAGUC driver in GDAL:
#  1) add "GDALRegister_ADAGUC();" between GMT and NetCDF drivers in /data/software/gdal-1.10.0/frmts/gdalallregister.cpp
#  2) add "void CPL_DLL GDALRegister_ADAGUC(void);" between GMT and NetCDF drivers in /data/software/gdal-1.10.0/gcore/gdal_frmts.h 

# Compile GDAL
cd /data/software/gdal-1.10.0/
./configure --prefix=/data/build LIBS="-ludunits2"

make
make install

# The environment has now been setup completely for the ADAGUC GDAL installation #

To query a netcdf file you can do:

# Determine which subdatasets are available
gdalinfo INTER_OPER_R___TAVGD___L3__19660101T000000_19660102T000000_0003.nc
# Get info about the subdataset
gdalinfo ADAGUC:"INTER_OPER_R___TAVGD___L3__19660101T000000_19660102T000000_0003.nc":prediction
#Convert this subdataset
gdal_translate -of AAIGRID ADAGUC:"INTER_OPER_R___TAVGD___L3__19660101T000000_19660102T000000_0003.nc":prediction INTER_OPER_R___TAVGD___L3__19660101T000000_19660102T000000_0003_asc.grd

Last modified 5 years ago Last modified on Jul 26, 2013 1:31:54 AM

Attachments (7)

Download all attachments as: .zip