Opened 13 years ago

Closed 12 years ago

#1485 closed defect (fixed)

Problem opening many HDF subdatasets at once

Reported by: pgao@… Owned by: warmerdam
Priority: normal Milestone: 1.4.2
Component: GDAL_Raster Version: 1.4.0
Severity: normal Keywords: hdf
Cc:

Description (last modified by warmerdam)

I run into a problem where if I open many subdatasets (~10), HDF driver fails right way.

If I open less than many (~10), the driver succeeds first, and but fails later when some opened subdatasets are closed, and some new ones are opened (the total is less than 10).

Change History (5)

comment:1 Changed 13 years ago by warmerdam

Description: modified (diff)
Milestone: 1.4.1
Priority: highestnormal
Severity: blockernormal
Status: newassigned

I'm going to dig into this today...

comment:2 Changed 13 years ago by warmerdam

I have been unable to reproduce this problem. I can run the following script opening and closing subdatasets, on both windows and linux and it works fine. If I change it to not actually close the subdatasets it fails after 30 subdatasets are opened (due to the HDF library handle limit). I don't see any inappropriate behavior.

#!/usr/bin/env python

import gdal

sds = [ 'HDF4_EOS:EOS_SWATH:"MOD28L2.A2001213.1525.004.2002197060500.hdf":Swath:sst',
        'HDF4_EOS:EOS_SWATH:"MOD28L2.A2001213.1525.004.2002197060500.hdf":Swath:sst4',
        'HDF4_EOS:EOS_SWATH:"MOD28L2.A2001213.1525.004.2002197060500.hdf":Swath:cldmsk_flags',
        'HDF4_EOS:EOS_SWATH:"MOD28L2.A2001213.1525.004.2002197060500.hdf":Swath:common_flags',
        'HDF4_EOS:EOS_SWATH:"MOD28L2.A2001213.1525.004.2002197060500.hdf":Swath:L2_flags',
        'HDF4_EOS:EOS_SWATH:"MOD28L2.A2001213.1525.004.2002197060500.hdf":Swath:quality' ]

y = []
for iter in range(0,100):
    x = [] 
    for i in range(0,len(sds)):
        x.append(gdal.Open( sds[i] ))

Actually, I think it would be nice for open hdf subdatasets to share a handle on the base file when there are multiple subdatasets open on the same base file. But such an improvement would need to be done outside the 1.4 branch I think. But this doesn't seem to be the problem described above. PS. I also ran the script under valgrind, and there was no sign of leaking file pointers.

comment:3 Changed 13 years ago by warmerdam

Milestone: 1.4.11.4.2

Migrated to 1.4.2 since we are waiting for feedback on the problem. If we get info to reproduce this in time for 1.4.1 we have push it in fast, but there is no point to hold 1.4.1 for it.

comment:4 Changed 13 years ago by warmerdam

Keywords: hdf added

I found a serious leak for HDF4 datasets that have no swaths (such as grid files for instance). The fix is in branches/1.4 as r11392, and will be incorporated in trunk when the Identify related fixes go in.

comment:5 Changed 12 years ago by warmerdam

Resolution: fixed
Status: assignedclosed

I'm closing under the assumption the problem fixed was the core issue here. Please reopen if you find otherwise.

Note: See TracTickets for help on using tickets.