Opened 14 years ago

Closed 5 years ago

#3466 closed enhancement (wontfix)

[PATCH] New Driver HDF4 HDFEOS:EOS_GRID

Reported by: vanbogaert Owned by: warmerdam
Priority: normal Milestone: closed_because_of_github_migration
Component: GDAL_Raster Version: unspecified
Severity: normal Keywords:
Cc: dron

Description

Give the ability to open and update all fields in a grid
and Create a HDFEOS:EOS_GRID via CreateCopy:

example:
gdal_translate -of HE -co COMPRESS=DEFLATE -co BLOCKXSIZE=256 -co BLOCKYSIZE=256  input.tif output.hdf:Grid_test

Note:
for tiling prupose in HDFEOS it needed that:
(RASTERXSIZE % BLOCKXSIZE) == 0 and (RASTERYSIZE % BLOCKYSIZE) == 0

Add SRS attribute for GCTP_GEO because loss datum in HDFEOS (no SphereCode in GCTP_GEO)

fieldname are setup via Band METADATA: BAND=name
NoData are setup via Band METADATA: NoData=value
Compress are setup via Band METADATA: COMPRESS=DEFLATE (actualy only DEFLATE)

Attachments (7)

hdfeos.patch (1.5 KB ) - added by vanbogaert 14 years ago.
Patch for install in Gdal
hdfeos.2.patch (1.5 KB ) - added by vanbogaert 14 years ago.
Patch for install in Gdal
hdf4compat.h (2.0 KB ) - added by vanbogaert 14 years ago.
Include file
he_dataset.h (4.8 KB ) - added by vanbogaert 14 years ago.
Include file
he_dataset.cpp (53.5 KB ) - added by vanbogaert 14 years ago.
Code
frmt_hdf4.html (21.8 KB ) - added by vanbogaert 14 years ago.
Documentation
hdf4imagedataset.cpp (187.5 KB ) - added by vanbogaert 14 years ago.
HDF4ImageDataset

Download all attachments as: .zip

Change History (18)

by vanbogaert, 14 years ago

Attachment: hdfeos.patch added

Patch for install in Gdal

by vanbogaert, 14 years ago

Attachment: hdfeos.2.patch added

Patch for install in Gdal

by vanbogaert, 14 years ago

Attachment: hdf4compat.h added

Include file

by vanbogaert, 14 years ago

Attachment: he_dataset.h added

Include file

by vanbogaert, 14 years ago

Attachment: he_dataset.cpp added

Code

comment:1 by vanbogaert, 14 years ago

Don't keep hdf4compat.h and hdfeos.2.patch (My mistake)

comment:2 by Even Rouault, 14 years ago

Eric,

I'm wondering why you didn't choose improving/fixing the HDF4ImageDataset driver instead of creating a fresh new one (*) ? I see there's a bit of code duplication between your new HE driver and code in hdf4dataset.cpp + the fact that it isn't very user friendly to have 2 ways of opening an EOS_GRID dataset (the current one with HDF4_EOS:EOS_GRID prefix and the new one you propose HDFEOS:EOS_GRID). I'd imagine that the EOS_GRID format could be selected as an option of CreateCopy(), such as PROFILE=EOS_GRID. If not specified, it would default to the GDAL HDF4 profile.

(*) : I guess one reason is the complexity of the driver... Perhaps subclassing HDF4ImageDataset/HDF4ImageRasterBand as HDF4EOSGRIDDataset/HDF4EOSGRIDRasterBand and overriding a few methods could be a more convenient way to add your changes into the driver ?

comment:3 by vanbogaert, 14 years ago

Dear Even,

Yes one reasons was the complexity of the existing driver but it support many features :) The existing HDF4ImageDataset/HDF4ImageRasterBand are more oriented to hdf4/field and support 2D and 3D. I'could give my advice about HDFEOS in a private email (eric.vanbogaert@…) ;-) But this format is used by NASA and ESA for public distribution. By the way, when LPDAAC(NASA) create a HDFEOS GRID it is probably via the HDF4/SDS interface and add METADATA for HDFEOS support:

for example:

  • Using GDgetfillvalue are ok when using GDsetfillvalue but with NASA EOS product it's needed to get SDreadattr.
  • With HDFEOS it is not possible to define a tile size that not respect (RASTERXSIZE % BLOCKXSIZE) == 0 and (RASTERYSIZE % BLOCKYSIZE) == 0, You have find this in MOD09GA in the grid 1km. But with the HDF4/SDS it is possible.

I'am sure that many people are happy with the actual implentation of the actual HDF4_EOS:EOS_GRID

Why this driver:

-DataCenter that produce data for Remote sensing use HDF4 and HDFEOS, but don't use COMPRESS feature (little strange with HDFEOS) and don't like to provide GeoTiff formats( more user frendly). In many case the processing chain is based only on HDF4 and HDFEOS. -It's so happy to multiply ours storage capacity by a simple gdal_translate -co COMPRESS=DEFLATE -When open a Grid, having acces to all 2D fields is also more easy.

But I don't think that HDFEOS have a big futur, HDFEOS5 exist now from many years and don't be very used. The best format is probably based on HDF5 SDSInterface using groups for define GRID and attributs like SRS and GeoTransform (GTiff support only one type for all fields..). Is it a needed formats? Probably to many years to become a standart...

When I works with "granules" from MODIS or MERIS, i need to make a mosaic for having a global representation of the data. If I made a VRT file, there are to mutch files and performances are very poor. With HDFEOS:EOS_GRID I have a other driver (actually in developpement) for acceding global data in shared mode for read, write and auto create HDFEOS file if granule is absent. With this a processing chain gain in more simple lines of code and quality.

Why create copy:

Setting projection parameter on th fly. Probably the best way is to works like NASA: SDS interface for all and setting the HDFEOS parameters when closing the dataset.

best regards,

Eric

comment:4 by vanbogaert, 14 years ago

RE: Perhaps subclassing HDF4ImageDataset/HDF4ImageRasterBand as HDF4EOSGRIDDataset/HDF4EOSGRIDRasterBand

Dear Even

Probably, initialy I woul'd like to support HDFEOS5.1 then it's not only for HDF4

By the way GDAL could'be support named band in the futur ?

comment:5 by Even Rouault, 14 years ago

Cc: dron added
Summary: New Driver HDF4 HDFEOS:EOS_GRID[PATCH] New Driver HDF4 HDFEOS:EOS_GRID

Luc, clearly, your comments show that I don't know enough with HDF4/HDFEOS to really understand what is at stake ;-) So I happily leave other potentially interested GDAL developers to deal with you. CC'ing Andrey Kiselev, the original author of the driver in case he might want to follow up.

comment:6 by dron, 14 years ago

Eric,

I am understand your reasons to create the separate driver for the particular HDF flavor, but I should note that it is not easy task to maintain the code as soon as it will be integrated in GDAL. The current driver is complex and messy, it contains many workarounds and hardcoded tricks for particular data products, it is a maintainer's headache, but for this price we have more or less flexible solution which works for most popular products quite well. The separate driver will not reduce maintainer's workload, it will add to this complexity instead. From what you wrote I think that the datasets of your interest could be handled with existing driver, we just need to add some more of those tricks. Can you consider improving the old HDF4 driver? Personally I am on position that there should not be alternate drivers for the same format without serious reasons, especially for such messy thing as HDF.

Best regards, Andrey

in reply to:  6 comment:7 by vanbogaert, 14 years ago

Replying to dron:

[skip...] Can you consider improving the old HDF4 driver?

Dear Andrey,

Yes, it's a way.

Adding multi-fields acces for HDF4_EOS:EOS_GRID when it's only 2D rank.
Adding CHUNK and COMPRESS in Create() HDF4Image
Adding Vgroup (by option) to HDF4Image (then SubDatase like HDFEOS )
and if possible, Adding HDFEOS MetaData by option to HDF4Image

Yours suggestion and comments are welcome :)

Best regards, Eric

comment:8 by vanbogaert, 14 years ago

Working progress request for comments

Add alternative open HDF4:

gdalinfo HDF4_SDS:EOS_GRID:MOD09GA.A2010055.h17v04.005.2010058075104.hdf:MODIS_Grid_1km_2D

View all fields in grid MODIS_Grid_1km_2D

Add alternative create HDF4:

gdal_translate -co COMPRESS=DEFLATE -co TILED=YES -co INTERLEAVE=BAND -of HDF4_GDAL -co GRIDNAME=SPOTVGT/GLOBAL/1km example.tif test_02.hdf
gdalinfo HDF4_SDS:GDAL_GRID:test_02.hdf

Actually no modification of HDF4Dataset

by vanbogaert, 14 years ago

Attachment: frmt_hdf4.html added

Documentation

by vanbogaert, 14 years ago

Attachment: hdf4imagedataset.cpp added

HDF4ImageDataset

comment:9 by Jukka Rahkonen, 9 years ago

Does this ticket handle this driver: http://www.gdal.org/frmt_hdf4.html?

If it does I think that that this ticket could be closed as fixed.

comment:10 by Even Rouault, 9 years ago

From a very quick skim, this ticket contains changes not yet applied

comment:11 by Even Rouault, 5 years ago

Milestone: closed_because_of_github_migration
Resolution: wontfix
Status: newclosed

This ticket has been automatically closed because Trac is no longer used for GDAL bug tracking, since the project has migrated to GitHub. If you believe this ticket is still valid, you may file it to https://github.com/OSGeo/gdal/issues if it is not already reported there.

Note: See TracTickets for help on using tickets.