gdalwarp does not properly close datasets in error conditions, can output errors in shared library unloading

[hsumanto]

I have two netcdf files containing multiple datasets. The netCDF driver produces some HDF5-DIAG error message on the first file but on the second file, the driver handles it without any error.

gdalwarp -of netCDF multipledataset1.nc dataset1.nc
Warning 1: No UNIDATA NC_GLOBAL:Conventions attribute 
Input file multipledataset1.nc has no raster bands.
HDF5-DIAG: Error detected in HDF5 (1.8.8) thread 0:
  #000: H5T.c line 1721 in H5Tclose(): not a datatype
    major: Invalid arguments to routine
    minor: Inappropriate type

gdalwarp -of netCDF multipledataset2.nc dataset2.nc2
Input file multipledataset2.nc has no raster bands.

Previous summary was "HDF5-DIAG Error detected when processing netcdf file with multiple datasets"

[hsumanto]

sample dataset used

[etourigny]

GDAL treats both files as having subdatasets (with no bands), therefore gdalwarp cannot operate on them, you have to work with the subdatasets.

The error for the first file, as well as the debugging output (GDAL: force close of multipledataset2.nc in GDALDriverManager cleanup.) is cause for minor concern, I will see what is wrong. It seems the input file is not closed properly, and is unrelated to the lack of support for gdalwarp with multiple subdatasets.

One way or another, don't count on having the fundamental problem fixed soon though (see my email to the list today in answer to Hendy's post).

$ gdalwarp --debug on -of netCDF multipledataset1.nc dataset1.nc
GDAL_netCDF: 
=====
calling nc_open( multipledataset1.nc )

GDAL_netCDF: driver detected file type=3, libnetcdf detected type=4
GDAL_netCDF: seting file type to 4, was 3
GDAL_netCDF: dim_count = 2
Warning 1: No UNIDATA NC_GLOBAL:Conventions attribute
GDAL_netCDF: var_count = 4
GDAL: GDALOpen(multipledataset1.nc, this=0x17656a0) succeeds as netCDF.
Input file multipledataset1.nc has no raster bands.
GDAL: In GDALDestroy - unloading GDAL shared library.
GDAL: force close of multipledataset1.nc in GDALDriverManager cleanup.
HDF5-DIAG: Error detected in HDF5 (1.8.8) thread 0:
  #000: H5T.c line 1721 in H5Tclose(): not a datatype
    major: Invalid arguments to routine
    minor: Inappropriate type
GDAL: GDALClose(multipledataset1.nc, this=0x17656a0)

 $ gdalwarp --debug on -of netCDF multipledataset2.nc dataset2.nc2
GDAL_netCDF: 
=====
calling nc_open( multipledataset2.nc )

GDAL_netCDF: driver detected file type=1, libnetcdf detected type=1
GDAL_netCDF: dim_count = 2
GDAL_netCDF: var_count = 9
GDAL_netCDF: variable #7 [lat] was ignored
GDAL_netCDF: variable #8 [lon] was ignored
GDAL: GDALOpen(multipledataset2.nc, this=0x12666a0) succeeds as netCDF.
Input file multipledataset2.nc has no raster bands.
GDAL: In GDALDestroy - unloading GDAL shared library.
GDAL: force close of multipledataset2.nc in GDALDriverManager cleanup.
GDAL: GDALClose(multipledataset2.nc, this=0x12666a0)

[hsumanto]

Thanks for having a look at the cause of the HDF5-Diag error, Etienne.

[etourigny]

With much help from Even and Frank, the source of the error message has been found.

It seems that the hdf5 library uses atextit() exit handlers (specifically the function H5_term_library()) which get called before gdal unloads. The call to nc_close (ehich calls H5close()) raises an error because the hdf5 library has been unloaded.

Disabling most of gdaldllmain.cpp seems to solve it temporarily, but a better solution should be found.

//#ifdef __GNUC__
#ifdef notdef

Not sure if this happens in Windows - adding Joaquim in cc... Can you see if you get the same error?

[jluis]

Replying to etourigny:

With much help from Even and Frank, the source of the error message has been found. It seems that the hdf5 library uses atextit() exit handlers (specifically the function H5_term_library()) which get called before gdal unloads. The call to nc_close (ehich calls H5close()) raises an error because the hdf5 library has been unloaded. Disabling most of gdaldllmain.cpp seems to solve it temporarily, but a better solution should be found. {{{ //#ifdef GNUC #ifdef notdef }}} Not sure if this happens in Windows - adding Joaquim in cc... Can you see if you get the same error?

[jluis]

Etienne,

On Win7 64 with HDF5 1.8.8 and netCDF 4.1.3 I get this

c:\>gdalwarp -of netCDF multipledataset1.nc dataset1.nc
Warning 1: No UNIDATA NC_GLOBAL:Conventions attribute
Input file multipledataset1.nc has no raster bands.

c:\>gdalwarp -of netCDF multipledataset2.nc dataset1.nc
Input file multipledataset2.nc has no raster bands.

and with the -debug mode

c:\>gdalwarp --debug on -of netCDF multipledataset1.nc dataset1.nc
GDAL_netCDF:
=====
calling nc_open( multipledataset1.nc )

GDAL_netCDF: driver detected file type=3, libnetcdf detected type=4
GDAL_netCDF: seting file type to 4, was 3
GDAL_netCDF: dim_count = 2
Warning 1: No UNIDATA NC_GLOBAL:Conventions attribute
GDAL_netCDF: var_count = 4
GDAL: GDALOpen(multipledataset1.nc, this=00000000005B6660) succeeds as netCDF.
Input file multipledataset1.nc has no raster bands.
GDAL: force close of multipledataset1.nc in GDALDriverManager cleanup.
GDAL: GDALClose(multipledataset1.nc, this=00000000005B6660)

[etourigny]

This confirms that Windows shared library handling does not have the same problem as in linux. Thanks Joaquim!

[etourigny]

Attaching a patch which resolves the issue in gdalwarp: exit(n) is replaced by GDALExit(n), which closes GDAL (and possibly OGR) ressources.

static int GDALExit( int nCode )
{
  const char  *pszDebug = CPLGetConfigOption("CPL_DEBUG",NULL);
  if( pszDebug && (EQUAL(pszDebug,"ON") || EQUAL(pszDebug,"") ) )
  {  
    GDALDumpOpenDatasets( stderr );
    CPLDumpSharedList( NULL );
  }

  GDALDestroyDriverManager();

#ifdef OGR_ENABLED
  OGRCleanupAll();
#endif

  exit( nCode );
}

In my opinion this should be added to the c api (in trunk and possibly other branches), and should be advertised in the gdal docs. This is an easy way to make sure datasets are closed automatically on exit.

Perhaps there are other gdal apps which could benefit from this.

Not commiting yet, waiting on input from others.

[etourigny]

Fixed in trunk (r24213) and 1.9(r24214).