Opened 9 years ago

Closed 3 years ago

#4890 closed enhancement (wontfix)

PATCH: DODS driver can now read 3-D datasets, metadata, projection, remote NetCDF files

Reported by: korosov Owned by: warmerdam
Priority: normal Milestone: closed_because_of_github_migration
Component: GDAL_Raster Version: svn-trunk
Severity: normal Keywords: dods driver netcdf hdf patch
Cc: blaa_katt, etourigny, warmerdam

Description

Patch for the DODS driver (dodsdataset2.cpp) and DODS autotest (dods.py).

Major improvements

  • The new DODS driver can now read N-dimensional datasets. The N-D input matrix is represented as a sequence of 2D bands. E.g. a 3D datasets (3 x 100 x 200) is returned as three bands 200 x 100 pixels each, a 4D dataset (3 x 2 x 300 x 500) is returned as 6 bands 300 x 500 pixels.
  • DODS driver gets names of dimensions of each variable from the file and does not assume only lat/lon or x/y.
  • DODS driver provides location of the band in high order dimensions in the band attributes. E.g.: if a remote NetCDF file has 4D variable with dimensions [time][depth][y][x] the following attributes will appear in each 2D band fetched by the DODS driver:
        dods_coordinates=[3][0]
        dods_dimensions=[time][depth]
    

It shows that 'depth' is the third and 'time' is the fourth dimension. This band is the fourth layer on time and the first layer on depth dimension.

  • DODS driver can read data both from HDF or NetCDF remote files
  • DODS driver fetches all global and band metadata is from remote server
  • DODS driver reads projection information from the server
  • If only the filename is given in the URL all bands are retrieved from all subdatasets. If a 'subdataset' (variable) name is specified after '?' only band(s) from this subdataset are returned. If order of dimension is not specified at the end of the subdataset name in square brackets the CF-compliant order is assumed (e.g. [time][y][x]).
  • The new DODS autotest has correct URL of TOVS_MONTHLY_PM_8502_NF.HDF.Z
  • DODS autotest tries to read all bands from a file containing 2D and 3D subdatasets
  • DODS autotest tries to read one band from a 3D subdataset
  • DODS autotest tries to read metadata
  • DODS autotest tries to read projection from NetCDF file

Examples

gdalinfo http://disc1.sci.gsfc.nasa.gov/opendap/tovs/TOVSAMNF/1985/032/TOVS_MONTHLY_PM_8502_NF.HDF.Z
gdalinfo http://disc1.sci.gsfc.nasa.gov/opendap/tovs/TOVSAMNF/1985/032/TOVS_MONTHLY_PM_8502_NF.HDF.Z?Data-Set-11
gdalinfo http://disc1.sci.gsfc.nasa.gov/opendap/tovs/TOVSAMNF/1985/032/TOVS_MONTHLY_PM_8502_NF.HDF.Z?Data_Set_9
gdalinfo http://disc1.sci.gsfc.nasa.gov/opendap/tovs/TOVSAMNF/1985/032/TOVS_MONTHLY_PM_8502_NF.HDF.Z?Data_Set_9[3][x][y]

Comments

  • DODS and NetCDF drivers have much in common. In fact some of the parts were just copied from the NetCDF driver. Duplication of code is generally a very bad habit and therefore should be omitted. Inheritance of DODS from NetCDF sounds like a best approach. But we are limited in resource to implement that.
  • DODS driver supports HTTP-like URLS: the subdataset name is provided after '?'. Ideally it should be similar to NeCDF or HDF driver syntax (HDF4_EOS:EOS_SWATH:"FILENAME":MODIS_SWATH_Type_L1B:EV_1KM_RefSB).
  • DODS driver has posibility to specify order of dimensions. Thats what NetCDF and HDF drivers are missing.
  • The driver was largely updated by blaa_katt, autotest - by korosov.

Attachments (2)

patch.dodsdataset2.cpp (45.4 KB) - added by korosov 9 years ago.
patch.dods.py (6.1 KB) - added by korosov 9 years ago.

Download all attachments as: .zip

Change History (12)

Changed 9 years ago by korosov

Attachment: patch.dodsdataset2.cpp added

Changed 9 years ago by korosov

Attachment: patch.dods.py added

comment:1 Changed 9 years ago by Even Rouault

Cc: etourigny added

Etienne, CC'ing you as it seems close to netCDF, in case you have some interest in the DODS driver and would like to review and take care of the patch. (Some update of the doc would likely be needed)

comment:2 Changed 9 years ago by korosov

We've noticed a small mistake in the example:

gdalinfo http://disc1.sci.gsfc.nasa.gov/opendap/tovs/TOVSAMNF/1985/032/TOVS_MONTHLY_PM_8502_NF.HDF.Z?Data_Set_11

Thankx, knutfd!

comment:3 Changed 9 years ago by etourigny

korosov - did you test this patch on linux and windows?

I have to say that I have no experience with DODS, so I can't really make constructive comments, and I also don't have the time to test and review, sorry.

Just a general comment - last commit to the driver was 2 years ago. Netcdf driver has improved quite a lot since then, there might be a few things that could be ported.

I'm not sure how easy subclassing from netcdf driver would be though, seems the whole driver would have to be re-written.

Just a clarification - what is the DODS approach for projection, does it use or support CF grid_mapping?

It seems to me (by glancing at the patch) that grid_mapping is not used for projection info, which is not a good idea. See the netcdf driver for a better treatment of projection and geotransform as read from CF grid_mapping metadata. Admittedly in this case it would be beneficial to subclass netcdfdataset.

Also, does this address the problem you raised in bug #4886 ? I don't see any code that reads actual lat/lon arrays to get geotransform.

comment:4 Changed 9 years ago by warmerdam

Cc: warmerdam added

I have suggested to Anton that he become a commiter to maintain the DODS driver long term.

For general purpose lon/lat geolocation arrays, please keep in mind the geolocation array mechanism.

http://trac.osgeo.org/gdal/wiki/rfc4_geolocate

comment:5 in reply to:  3 Changed 9 years ago by korosov

Replying to etourigny:

korosov - did you test this patch on linux and windows?

Yes, both works good. Though on some linux machines gdal crashes when accessing aggregated datasets with more than 300 bands.

I have to say that I have no experience with DODS, so I can't really make constructive comments, and I also don't have the time to test and review, sorry.

Just a general comment - last commit to the driver was 2 years ago. Netcdf driver has improved quite a lot since then, there might be a few things that could be ported.

I'm not sure how easy subclassing from netcdf driver would be though, seems the whole driver would have to be re-written.

Just a clarification - what is the DODS approach for projection, does it use or support CF grid_mapping?

I should admit it is not as good as in the NetCDF driver. It is very limited in fact. It tries to take attribute 'spatial_ref' from the grid_mapping which is the WKT string. Obviously it exists only if NetDCF files at OpenDAP server were generated by GDAL. This update of the driver was mainly for reading 3D datasets. It is project related, and we aimed at limited functionality.

It seems to me (by glancing at the patch) that grid_mapping is not used for projection info, which is not a good idea. See the netcdf driver for a better treatment of projection and geotransform as read from CF grid_mapping metadata. Admittedly in this case it would be beneficial to subclass netcdfdataset.

Also, does this address the problem you raised in bug #4886 ? I don't see any code that reads actual lat/lon arrays to get geotransform.

We don't read GeoTransform? to ovoid the problem of #4886. Arrays with dimension variables are read for calculating GeoTranfroms? at lines 1215 - 1218. However 'lat' or 'lon' or 'x' or 'y' are not hardcoded anymore, rather the name of the dimension variable is read from the attributes of the dataset.

comment:6 Changed 8 years ago by etourigny

korosov - sorry I didn't look into this... Are you willing to become a gdal maintainer for this driver? If not, could you provide some tests (in the autotest/gdrivers/) that these changes would pass?

comment:7 Changed 7 years ago by Jukka Rahkonen

korosov - did you get frightened by the suggestion to become a maintainer? How do you feel now 21 months later?

comment:8 Changed 7 years ago by korosov

jratike80 I'm not frightened :) I'll be willing to contribute in that way! What does it imply from my side to be maintainer? I need to assess available resources.

comment:9 Changed 7 years ago by Jukka Rahkonen

Read first this document http://trac.osgeo.org/gdal/wiki/HowToContribute and ask more advice from the gdal-dev mailing list.

comment:10 Changed 3 years ago by Even Rouault

Milestone: closed_because_of_github_migration
Resolution: wontfix
Status: newclosed

This ticket has been automatically closed because Trac is no longer used for GDAL bug tracking, since the project has migrated to GitHub?. If you believe this ticket is still valid, you may file it to https://github.com/OSGeo/gdal/issues if it is not already reported there.

Note: See TracTickets for help on using tickets.