Problem opening many HDF subdatasets at once

[pgao@…]

I run into a problem where if I open many subdatasets (~10), HDF driver fails right way.

If I open less than many (~10), the driver succeeds first, and but fails later when some opened subdatasets are closed, and some new ones are opened (the total is less than 10).

[warmerdam]

I'm going to dig into this today...

[warmerdam]

I have been unable to reproduce this problem. I can run the following script opening and closing subdatasets, on both windows and linux and it works fine. If I change it to not actually close the subdatasets it fails after 30 subdatasets are opened (due to the HDF library handle limit). I don't see any inappropriate behavior.

#!/usr/bin/env python

import gdal

sds = [ 'HDF4_EOS:EOS_SWATH:"MOD28L2.A2001213.1525.004.2002197060500.hdf":Swath:sst',
        'HDF4_EOS:EOS_SWATH:"MOD28L2.A2001213.1525.004.2002197060500.hdf":Swath:sst4',
        'HDF4_EOS:EOS_SWATH:"MOD28L2.A2001213.1525.004.2002197060500.hdf":Swath:cldmsk_flags',
        'HDF4_EOS:EOS_SWATH:"MOD28L2.A2001213.1525.004.2002197060500.hdf":Swath:common_flags',
        'HDF4_EOS:EOS_SWATH:"MOD28L2.A2001213.1525.004.2002197060500.hdf":Swath:L2_flags',
        'HDF4_EOS:EOS_SWATH:"MOD28L2.A2001213.1525.004.2002197060500.hdf":Swath:quality' ]

y = []
for iter in range(0,100):
    x = [] 
    for i in range(0,len(sds)):
        x.append(gdal.Open( sds[i] ))

Actually, I think it would be nice for open hdf subdatasets to share a handle on the base file when there are multiple subdatasets open on the same base file. But such an improvement would need to be done outside the 1.4 branch I think. But this doesn't seem to be the problem described above. PS. I also ran the script under valgrind, and there was no sign of leaking file pointers.

[warmerdam]

Migrated to 1.4.2 since we are waiting for feedback on the problem. If we get info to reproduce this in time for 1.4.1 we have push it in fast, but there is no point to hold 1.4.1 for it.

[warmerdam]

I found a serious leak for HDF4 datasets that have no swaths (such as grid files for instance). The fix is in branches/1.4 as r11392, and will be incorporated in trunk when the Identify related fixes go in.

[warmerdam]

I'm closing under the assumption the problem fixed was the core issue here. Please reopen if you find otherwise.