memory leak in hdf driver

[vincentschut]

I'm afraid I've encountered a memory leak in the hdf driver. Context: because of massive parallel (cluster) processing, I'm reusing a python instance for lots of jobs. Some of these jobs use gdal to read and/or save data. In some cases, I saw the memory use of my python worker processes grow with each job, until the whole lot got killed. I've got 8 gig on that server :-) . I've been able to track down the leak to the use of gdal.Open on a hdf (modis) dataset. I guess this means that either the gdal hdf driver or libhdf4 has the leak.

Here is some python code (runs on linux only due to the code to get the mem usage) to prove a leak when opening a hdf file, while no leak when opening a tif. Run in a folder with 'data.hdf' and 'data.tif' present, or change those names to an existing hdf and tif file.

Using 64bit linux, gdal svn rev. 17228 (today), libhdf4.2r2, python 2.6.2

========= python code =========

import os
from osgeo import gdal

def getmemory():
   proc_status = '/proc/%d/status' % os.getpid()
   scale = {'kB': 1024.0, 'mB': 1024.0*1024.0,
         'KB': 1024.0, 'MB': 1024.0*1024.0}
   v = open(proc_status).read()
   i = v.index('VmSize:')
   v = v[i:].split(None, 3)
   return (int(v[1]) * scale[v[2]])

nFiles = 100

m0 = getmemory()
print 'memory usage before:', m0
print
print nFiles, 'times the same hdf file'
for i in range(nFiles):
   gdal.OpenShared('data.hdf')

m1 = getmemory()
print 'memory usage now:', m1, '  difference:', m1-m0
print
print nFiles, 'times the same tif file'
for i in range(nFiles):
   gdal.OpenShared('data.tif')

m2 = getmemory()
print 'memory usage now:', m2, '  difference:', m2-m1

[warmerdam]

I have tried running the command:

gdalinfo -mm HDF4_EOS:EOS_SWATH:"MOD28L2.A2001213.1525.004.2002197060500.hdf":Swath:sst

under valgrind and there is no apparent memory leaks according to valgrind. Are you working on linux? Can you try using valgrind to identify the leaks you are encountering?