Opened 12 years ago

Closed 10 years ago

#2208 closed defect (fixed)

Slow access to compressed HDF4

Reported by: nowakpl Owned by: warmerdam
Priority: normal Milestone: 1.8.0
Component: GDAL_Raster Version: svn-trunk
Severity: normal Keywords: HDF4
Cc:

Description

The problem is GDAL accessing the data one line at a time. I made a quick and dirty fix by setting nBlockYSize to 128 and some additional code. This is by no means a proper fix.

ftp://ladsweb.nascom.nasa.gov/allData/5/MOD02QKM/2008/038/MOD02QKM.A2008038.0210.005.2008038092447.hdf time /opt/gdal-r13204_x86-64_ubuntu7/bin/gdal_translate HDF4_EOS:EOS_SWATH:"MOD02QKM.A2008038.0210.005.2008038092447.hdf":MODIS_SWATH_Type_L1B:EV_250_RefSB test.tiff Input file size is 5416, 8120 0...10...20...30...40...50...60...70...80...90...100 - done. real 2m53.651s user 2m50.450s

Attachments (1)

hdf4.patch (6.6 KB) - added by nowakpl 12 years ago.

Download all attachments as: .zip

Change History (5)

Changed 12 years ago by nowakpl

Attachment: hdf4.patch added

comment:1 Changed 12 years ago by warmerdam

Component: defaultGDAL_Raster
Keywords: HDF4 added
Milestone: 1.6.0
Resolution: fixed
Status: newclosed

I have encorporated a variation on this patch in trunk (r13785) but only operating on SDS datasets (all I can easily test just now). Given testing it could be easily applied to other cases, and likely back ported to 1.5 branch.

Thanks!

comment:2 Changed 12 years ago by jluis

Version: unspecifiedsvn-trunk

I'm reopening this because I encountered a very slow performance using the example file below. The comparisons are done against Matlab and a converted to geotiff version of the same file.

Joaquim Luis

The example file can be get from ftp://data.nodc.noaa.gov/pub/data.nodc/pathfinder/Version5.0/Monthly/1991/199101.s04m1pfv50-sst-16b.hdf

The timings below (in seconds) respect my MEX for doing gdal read inside Matlab. That program is called gdalread. Hardware is a 2.5 GHz core 2 duo

Let tt.hdf = 199101.s04m1pfv50-sst-16b.hdf

tic,Z = gdalread('c:\tt.hdf');toc elapsed_time = 21.5

However, using the internal Matlab solution one get:

tic,sst=hdfread('c:\tt.hdf', 'sst', 'index', {[1 1],[1 1], [4096 8192]});toc elapsed_time = 0.8120

Just to show that the problem is not in "gdalread" I converted the hdf file to getiff (lixo.tiff), which took about 25 sec

tic,Z = gdalread('c:\lixo.tiff');toc elapsed_time = 2.0930

The slow performance is noticeable when fetching only the file's metadata

tic,Z = gdalread('c:\tt.hdf','-M');toc elapsed_time = 1.0310

% Matlab's way tic,Z=hdfinfo('c:\tt.hdf');toc elapsed_time = 0.0310

% On the geotiff file tic,Z = gdalread('c:\lixo.tiff','-M');toc elapsed_time = 0.0310

comment:3 Changed 12 years ago by Even Rouault

Resolution: fixed
Status: closedreopened

comment:4 Changed 10 years ago by Even Rouault

Milestone: 1.6.41.8.0
Resolution: fixed
Status: reopenedclosed

r18964 /trunk/gdal/frmts/hdf4/hdf4imagedataset.cpp: HDF4 : speed up access to HDF4_SDS datasets; allow multi-line block dimension for HDF4_EOS datasets (#2208)

r18965 /trunk/autotest/gcore/hdf4_read.py: Add testing of HDF4 datasets available on http://download.osgeo.org/gdal/data/hdf4

r18966 /trunk/autotest/gcore/hdf4_read.py: HDF4 : Test faster access to 199101.s04m1pfv50-sst-16b.hdf (#2208)

Note: See TracTickets for help on using tickets.