Slow access to compressed HDF4

[nowakpl]

The problem is GDAL accessing the data one line at a time. I made a quick and dirty fix by setting nBlockYSize to 128 and some additional code. This is by no means a proper fix.

​ftp://ladsweb.nascom.nasa.gov/allData/5/MOD02QKM/2008/038/MOD02QKM.A2008038.0210.005.2008038092447.hdf time /opt/gdal-r13204_x86-64_ubuntu7/bin/gdal_translate HDF4_EOS:EOS_SWATH:"MOD02QKM.A2008038.0210.005.2008038092447.hdf":MODIS_SWATH_Type_L1B:EV_250_RefSB test.tiff Input file size is 5416, 8120 0...10...20...30...40...50...60...70...80...90...100 - done. real 2m53.651s user 2m50.450s

[warmerdam]

I have encorporated a variation on this patch in trunk (r13785) but only operating on SDS datasets (all I can easily test just now). Given testing it could be easily applied to other cases, and likely back ported to 1.5 branch.

Thanks!

[jluis]

I'm reopening this because I encountered a very slow performance using the example file below. The comparisons are done against Matlab and a converted to geotiff version of the same file.

Joaquim Luis

The example file can be get from ​ftp://data.nodc.noaa.gov/pub/data.nodc/pathfinder/Version5.0/Monthly/1991/199101.s04m1pfv50-sst-16b.hdf

The timings below (in seconds) respect my MEX for doing gdal read inside Matlab. That program is called gdalread. Hardware is a 2.5 GHz core 2 duo

Let tt.hdf = 199101.s04m1pfv50-sst-16b.hdf

tic,Z = gdalread('c:\tt.hdf');toc elapsed_time = 21.5

However, using the internal Matlab solution one get:

tic,sst=hdfread('c:\tt.hdf', 'sst', 'index', {[1 1],[1 1], [4096 8192]});toc elapsed_time = 0.8120

Just to show that the problem is not in "gdalread" I converted the hdf file to getiff (lixo.tiff), which took about 25 sec

tic,Z = gdalread('c:\lixo.tiff');toc elapsed_time = 2.0930

The slow performance is noticeable when fetching only the file's metadata

tic,Z = gdalread('c:\tt.hdf','-M');toc elapsed_time = 1.0310

% Matlab's way tic,Z=hdfinfo('c:\tt.hdf');toc elapsed_time = 0.0310

% On the geotiff file tic,Z = gdalread('c:\lixo.tiff','-M');toc elapsed_time = 0.0310

[Even Rouault]

• I've enabled using multi-line block dimension for HDF4_EOS datasets. Tested with ​ftp://ladsweb.nascom.nasa.gov/allData/5/MOD02QKM/2008/038/MOD02QKM.A2008038.0135.005.2010025084041.hdf (the dataset mentionned in the ticket is no longer available)

• I've found a way of speeding up considerably the access to ​ftp://data.nodc.noaa.gov/pub/data.nodc/pathfinder/Version5.0/Monthly/1991/199101.s04m1pfv50-sst-16b.hdf by avoiding to do SDselect() / SDendaccess() for each block access, which triggers heavy zlib usage. Now SDselect() is done the first time and the identifier is cached at the dataset level.

r18964 /trunk/gdal/frmts/hdf4/hdf4imagedataset.cpp: HDF4 : speed up access to HDF4_SDS datasets; allow multi-line block dimension for HDF4_EOS datasets (#2208)

r18965 /trunk/autotest/gcore/hdf4_read.py: Add testing of HDF4 datasets available on ​http://download.osgeo.org/gdal/data/hdf4

r18966 /trunk/autotest/gcore/hdf4_read.py: HDF4 : Test faster access to 199101.s04m1pfv50-sst-16b.hdf (#2208)