HDF4 reading MISR Level 3 with incorrect dimension

[ilucena]

The first sub-dataset indicates the existence of 4 dimensions as shown by gdalinfo:

% gdalinfo MISR_AM1_CGAS_MAY_2007_F15_0031.hdf
...
Subdatasets:
  SUBDATASET_1_NAME=HDF4_EOS:EOS_GRID:"MISR_AM1_CGAS_MAY_2007_F15_0031.hdf":AerosolParameterAverage:Absorbing optical depth
  SUBDATASET_1_DESC=[360x720x9x4] Absorbing optical depth AerosolParameterAverage (32-bit floating-point)
...

As I translate the whole sub-dataset to GTIFF it creates 9 bands:

% gdal_translate "HDF4_EOS:EOS_GRID:MISR_AM1_CGAS_MAY_2007_F15_0031.hdf:AerosolParameterAverage:Absorbing optical depth" May_2007.tif -sds
Input file size is 720, 360
0...10...20...30...40...50...60...70...80...90...100 - done.

% gdalinfo May_2007.tif
...
Image Structure Metadata:
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  (-180.0000000,  90.0000000) (180d 0'0.00"W, 90d 0'0.00"N)
Lower Left  (-180.0000000, -90.0000000) (180d 0'0.00"W, 90d 0'0.00"S)
Upper Right ( 180.0000000,  90.0000000) (180d 0'0.00"E, 90d 0'0.00"N)
Lower Right ( 180.0000000, -90.0000000) (180d 0'0.00"E, 90d 0'0.00"S)
Center      (   0.0000000,   0.0000000) (  0d 0'0.01"E,  0d 0'0.01"N)
Band 1 Block=720x1 Type=Float32, ColorInterp=Gray
  NoData Value=-9999
Band 2 Block=720x1 Type=Float32, ColorInterp=Undefined
  NoData Value=-9999
Band 3 Block=720x1 Type=Float32, ColorInterp=Undefined
  NoData Value=-9999
Band 4 Block=720x1 Type=Float32, ColorInterp=Undefined
  NoData Value=-9999
Band 5 Block=720x1 Type=Float32, ColorInterp=Undefined
  NoData Value=-9999
Band 6 Block=720x1 Type=Float32, ColorInterp=Undefined
  NoData Value=-9999
Band 7 Block=720x1 Type=Float32, ColorInterp=Undefined
  NoData Value=-9999
Band 8 Block=720x1 Type=Float32, ColorInterp=Undefined
  NoData Value=-9999
Band 9 Block=720x1 Type=Float32, ColorInterp=Undefined
  NoData Value=-9999

But by looking at the image it seams like it got dimensions wrong. There are just vertical stripes along the image.

Sample dataset are available at:

​http://eosweb.larc.nasa.gov/PRODOCS/misr/level3/level3_CGAS_small.html

This particular one came from:

​ftp://l4ftl01.larc.nasa.gov/MISR/MIL3MAE.004/2007.05.01/

[ilucena]

OpenEV screenshot of GTIFF file

[Jukka Rahkonen]

I do not understand HDF4 at all but I have a feeling that some misunderstanding was also a reason for this ticket.

The dimensions of the selected subdataset are, as reported [360x720x9x4] GDAL decides that the two first dimensions are the X and Y sizes of the raster and the third dimension is telling the number of bands.

This theory can be verified by converting another subset "AerosolParameterAverage:Algorithm type count" [360x720x10x2] into GeoTIFF. It contains 10 bands.

I believe that the meaning of -sds parameter of gdal_translate is understood in a wrong way. It does not stand for "the whole subdataset" but "all the subdatasets". This is from gdal_tranlate manual page:

Copy all subdatasets of this file to individual output files. 
Use with formats like HDF or OGDI that have subdatasets. 
The output file naming scheme has changed in GDAL 1.11 
(e.g. ofile_1.tif, ofile_2.tif).

In the example command -sds has no effect because one subdataset is explicitly selected. I do not know how -sds should be used. I thought that this command would create 9 separate tiff files but it did not work for me:

gdal_translate "HDF4_EOS:EOS_GRID:MISR_AM1_CGAS_MAY_2007_F15_0031.hdf" -sds ofile.tif
GDALOpen failed - 0

I consider that there is not a bug as the title is expressing it. There may be some fuzzy logic in detecting the number of bands and missing features with 4 and more dimensions. It seems to me that user has no control on which dimension is used for splitting data into bands. That's probably worth a new ticket. Perhaps this NetCDF patch is having a similar aim #2540.

I keep this ticket open for some weeks and wait that ilucena or someone else who understands HDF4 better that I makes a new review. If there will be no activity I am planning to close this ticket as invalid.

[ilucena]

You may be right to say that the -sds parameter has no effect on my example. I don't think the -sds will work if the output format doesn't support subdatasets and it will not going to create several .tif files, as far as I know.

The HDF format is very flexible and in some cases the GDAL driver is able to identify the data provider/product to make the correct decision about the dimensions.

But as far as I know there is no way for a user to inform the HDF4 driver how they want the dimensions to be processed ​http://www.gdal.org/frmt_hdf4.html.

So maybe the solution is to follow the NetCDF example.

[Jukka Rahkonen]

Option -sds should create many tiff files. See http://trac.osgeo.org/gdal/ticket/5119. Perhaps is is somehow broken or all subdatasets should have similar dimensions etc. Do you have HDF files to test with?

[Even Rouault]

This ticket has been automatically closed because Trac is no longer used for GDAL bug tracking, since the project has migrated to GitHub. If you believe this ticket is still valid, you may file it to ​https://github.com/OSGeo/gdal/issues if it is not already reported there.