PATCH: DODS driver can now read 3-D datasets, metadata, projection, remote NetCDF files

[korosov]

Patch for the DODS driver (dodsdataset2.cpp) and DODS autotest (dods.py).

Major improvements

• The new DODS driver can now read N-dimensional datasets. The N-D input matrix is represented as a sequence of 2D bands. E.g. a 3D datasets (3 x 100 x 200) is returned as three bands 200 x 100 pixels each, a 4D dataset (3 x 2 x 300 x 500) is returned as 6 bands 300 x 500 pixels.

• DODS driver gets names of dimensions of each variable from the file and does not assume only lat/lon or x/y.

• DODS driver provides location of the band in high order dimensions in the band attributes. E.g.: if a remote NetCDF file has 4D variable with dimensions [time][depth][y][x] the following attributes will appear in each 2D band fetched by the DODS driver:

      dods_coordinates=[3][0]
      dods_dimensions=[time][depth]
  

It shows that 'depth' is the third and 'time' is the fourth dimension. This band is the fourth layer on time and the first layer on depth dimension.

• DODS driver can read data both from HDF or NetCDF remote files

• DODS driver fetches all global and band metadata is from remote server

• DODS driver reads projection information from the server

• If only the filename is given in the URL all bands are retrieved from all subdatasets. If a 'subdataset' (variable) name is specified after '?' only band(s) from this subdataset are returned. If order of dimension is not specified at the end of the subdataset name in square brackets the CF-compliant order is assumed (e.g. [time][y][x]).

• The new DODS autotest has correct URL of TOVS_MONTHLY_PM_8502_NF.HDF.Z

• DODS autotest tries to read all bands from a file containing 2D and 3D subdatasets

• DODS autotest tries to read one band from a 3D subdataset

• DODS autotest tries to read metadata

• DODS autotest tries to read projection from NetCDF file

Examples

gdalinfo http://disc1.sci.gsfc.nasa.gov/opendap/tovs/TOVSAMNF/1985/032/TOVS_MONTHLY_PM_8502_NF.HDF.Z
gdalinfo http://disc1.sci.gsfc.nasa.gov/opendap/tovs/TOVSAMNF/1985/032/TOVS_MONTHLY_PM_8502_NF.HDF.Z?Data-Set-11
gdalinfo http://disc1.sci.gsfc.nasa.gov/opendap/tovs/TOVSAMNF/1985/032/TOVS_MONTHLY_PM_8502_NF.HDF.Z?Data_Set_9
gdalinfo http://disc1.sci.gsfc.nasa.gov/opendap/tovs/TOVSAMNF/1985/032/TOVS_MONTHLY_PM_8502_NF.HDF.Z?Data_Set_9[3][x][y]

Comments

• DODS and NetCDF drivers have much in common. In fact some of the parts were just copied from the NetCDF driver. Duplication of code is generally a very bad habit and therefore should be omitted. Inheritance of DODS from NetCDF sounds like a best approach. But we are limited in resource to implement that.
• DODS driver supports HTTP-like URLS: the subdataset name is provided after '?'. Ideally it should be similar to NeCDF or HDF driver syntax (HDF4_EOS:EOS_SWATH:"FILENAME":MODIS_SWATH_Type_L1B:EV_1KM_RefSB).
• DODS driver has posibility to specify order of dimensions. Thats what NetCDF and HDF drivers are missing.
• The driver was largely updated by blaa_katt, autotest - by korosov.

[Even Rouault]

Etienne, CC'ing you as it seems close to netCDF, in case you have some interest in the DODS driver and would like to review and take care of the patch. (Some update of the doc would likely be needed)

[korosov]

We've noticed a small mistake in the example:

gdalinfo http://disc1.sci.gsfc.nasa.gov/opendap/tovs/TOVSAMNF/1985/032/TOVS_MONTHLY_PM_8502_NF.HDF.Z?Data_Set_11

Thankx, knutfd!

[etourigny]

korosov - did you test this patch on linux and windows?

I have to say that I have no experience with DODS, so I can't really make constructive comments, and I also don't have the time to test and review, sorry.

Just a general comment - last commit to the driver was 2 years ago. Netcdf driver has improved quite a lot since then, there might be a few things that could be ported.

I'm not sure how easy subclassing from netcdf driver would be though, seems the whole driver would have to be re-written.

Just a clarification - what is the DODS approach for projection, does it use or support CF grid_mapping?

It seems to me (by glancing at the patch) that grid_mapping is not used for projection info, which is not a good idea. See the netcdf driver for a better treatment of projection and geotransform as read from CF grid_mapping metadata. Admittedly in this case it would be beneficial to subclass netcdfdataset.

Also, does this address the problem you raised in bug #4886 ? I don't see any code that reads actual lat/lon arrays to get geotransform.

[warmerdam]

I have suggested to Anton that he become a commiter to maintain the DODS driver long term.

For general purpose lon/lat geolocation arrays, please keep in mind the geolocation array mechanism.

http://trac.osgeo.org/gdal/wiki/rfc4_geolocate

[korosov]

Replying to etourigny:

korosov - did you test this patch on linux and windows?

Yes, both works good. Though on some linux machines gdal crashes when accessing aggregated datasets with more than 300 bands.

I have to say that I have no experience with DODS, so I can't really make constructive comments, and I also don't have the time to test and review, sorry.

Just a general comment - last commit to the driver was 2 years ago. Netcdf driver has improved quite a lot since then, there might be a few things that could be ported.

I'm not sure how easy subclassing from netcdf driver would be though, seems the whole driver would have to be re-written.

Just a clarification - what is the DODS approach for projection, does it use or support CF grid_mapping?

I should admit it is not as good as in the NetCDF driver. It is very limited in fact. It tries to take attribute 'spatial_ref' from the grid_mapping which is the WKT string. Obviously it exists only if NetDCF files at OpenDAP server were generated by GDAL. This update of the driver was mainly for reading 3D datasets. It is project related, and we aimed at limited functionality.

It seems to me (by glancing at the patch) that grid_mapping is not used for projection info, which is not a good idea. See the netcdf driver for a better treatment of projection and geotransform as read from CF grid_mapping metadata. Admittedly in this case it would be beneficial to subclass netcdfdataset.

Also, does this address the problem you raised in bug #4886 ? I don't see any code that reads actual lat/lon arrays to get geotransform.

We don't read GeoTransform to ovoid the problem of #4886. Arrays with dimension variables are read for calculating GeoTranfroms at lines 1215 - 1218. However 'lat' or 'lon' or 'x' or 'y' are not hardcoded anymore, rather the name of the dimension variable is read from the attributes of the dataset.

[etourigny]

korosov - sorry I didn't look into this... Are you willing to become a gdal maintainer for this driver? If not, could you provide some tests (in the autotest/gdrivers/) that these changes would pass?

[Jukka Rahkonen]

korosov - did you get frightened by the suggestion to become a maintainer? How do you feel now 21 months later?

[korosov]

jratike80 I'm not frightened :) I'll be willing to contribute in that way! What does it imply from my side to be maintainer? I need to assess available resources.

[Jukka Rahkonen]

Read first this document http://trac.osgeo.org/gdal/wiki/HowToContribute and ask more advice from the gdal-dev mailing list.

[Even Rouault]

This ticket has been automatically closed because Trac is no longer used for GDAL bug tracking, since the project has migrated to GitHub. If you believe this ticket is still valid, you may file it to ​https://github.com/OSGeo/gdal/issues if it is not already reported there.