Opened 7 years ago

Closed 7 years ago

#4554 closed defect (fixed)

gdalwarp does not properly close datasets in error conditions, can output errors in shared library unloading

Reported by: hsumanto Owned by: warmerdam
Priority: normal Milestone:
Component: Utilities Version: unspecified
Severity: normal Keywords: netcdf
Cc: etourigny, jluis, Even Rouault, warmerdam

Description (last modified by etourigny)

I have two netcdf files containing multiple datasets. The netCDF driver produces some HDF5-DIAG error message on the first file but on the second file, the driver handles it without any error.

gdalwarp -of netCDF multipledataset1.nc dataset1.nc
Warning 1: No UNIDATA NC_GLOBAL:Conventions attribute 
Input file multipledataset1.nc has no raster bands.
HDF5-DIAG: Error detected in HDF5 (1.8.8) thread 0:
  #000: H5T.c line 1721 in H5Tclose(): not a datatype
    major: Invalid arguments to routine
    minor: Inappropriate type

gdalwarp -of netCDF multipledataset2.nc dataset2.nc2
Input file multipledataset2.nc has no raster bands.

Previous summary was "HDF5-DIAG Error detected when processing netcdf file with multiple datasets"

Attachments (1)

dataset.zip (2.3 KB) - added by hsumanto 7 years ago.
sample dataset used

Download all attachments as: .zip

Change History (11)

Changed 7 years ago by hsumanto

Attachment: dataset.zip added

sample dataset used

comment:1 Changed 7 years ago by Even Rouault

Cc: etourigny added

comment:2 Changed 7 years ago by etourigny

GDAL treats both files as having subdatasets (with no bands), therefore gdalwarp cannot operate on them, you have to work with the subdatasets.

The error for the first file, as well as the debugging output (GDAL: force close of multipledataset2.nc in GDALDriverManager cleanup.) is cause for minor concern, I will see what is wrong. It seems the input file is not closed properly, and is unrelated to the lack of support for gdalwarp with multiple subdatasets.

One way or another, don't count on having the fundamental problem fixed soon though (see my email to the list today in answer to Hendy's post).

$ gdalwarp --debug on -of netCDF multipledataset1.nc dataset1.nc
GDAL_netCDF: 
=====
calling nc_open( multipledataset1.nc )

GDAL_netCDF: driver detected file type=3, libnetcdf detected type=4
GDAL_netCDF: seting file type to 4, was 3
GDAL_netCDF: dim_count = 2
Warning 1: No UNIDATA NC_GLOBAL:Conventions attribute
GDAL_netCDF: var_count = 4
GDAL: GDALOpen(multipledataset1.nc, this=0x17656a0) succeeds as netCDF.
Input file multipledataset1.nc has no raster bands.
GDAL: In GDALDestroy - unloading GDAL shared library.
GDAL: force close of multipledataset1.nc in GDALDriverManager cleanup.
HDF5-DIAG: Error detected in HDF5 (1.8.8) thread 0:
  #000: H5T.c line 1721 in H5Tclose(): not a datatype
    major: Invalid arguments to routine
    minor: Inappropriate type
GDAL: GDALClose(multipledataset1.nc, this=0x17656a0)

 $ gdalwarp --debug on -of netCDF multipledataset2.nc dataset2.nc2
GDAL_netCDF: 
=====
calling nc_open( multipledataset2.nc )

GDAL_netCDF: driver detected file type=1, libnetcdf detected type=1
GDAL_netCDF: dim_count = 2
GDAL_netCDF: var_count = 9
GDAL_netCDF: variable #7 [lat] was ignored
GDAL_netCDF: variable #8 [lon] was ignored
GDAL: GDALOpen(multipledataset2.nc, this=0x12666a0) succeeds as netCDF.
Input file multipledataset2.nc has no raster bands.
GDAL: In GDALDestroy - unloading GDAL shared library.
GDAL: force close of multipledataset2.nc in GDALDriverManager cleanup.
GDAL: GDALClose(multipledataset2.nc, this=0x12666a0)

comment:3 Changed 7 years ago by hsumanto

Thanks for having a look at the cause of the HDF5-Diag error, Etienne.

comment:4 Changed 7 years ago by etourigny

Cc: jluis added

With much help from Even and Frank, the source of the error message has been found.

It seems that the hdf5 library uses atextit() exit handlers (specifically the function H5_term_library()) which get called before gdal unloads. The call to nc_close (ehich calls H5close()) raises an error because the hdf5 library has been unloaded.

Disabling most of gdaldllmain.cpp seems to solve it temporarily, but a better solution should be found.

//#ifdef __GNUC__
#ifdef notdef

Not sure if this happens in Windows - adding Joaquim in cc... Can you see if you get the same error?

comment:5 in reply to:  4 Changed 7 years ago by jluis

Description: modified (diff)

Replying to etourigny:

With much help from Even and Frank, the source of the error message has been found.

It seems that the hdf5 library uses atextit() exit handlers (specifically the function H5_term_library()) which get called before gdal unloads. The call to nc_close (ehich calls H5close()) raises an error because the hdf5 library has been unloaded.

Disabling most of gdaldllmain.cpp seems to solve it temporarily, but a better solution should be found.

//#ifdef __GNUC__
#ifdef notdef

Not sure if this happens in Windows - adding Joaquim in cc... Can you see if you get the same error?

comment:6 Changed 7 years ago by jluis

Description: modified (diff)

Etienne,

On Win7 64 with HDF5 1.8.8 and netCDF 4.1.3 I get this

c:\>gdalwarp -of netCDF multipledataset1.nc dataset1.nc
Warning 1: No UNIDATA NC_GLOBAL:Conventions attribute
Input file multipledataset1.nc has no raster bands.

c:\>gdalwarp -of netCDF multipledataset2.nc dataset1.nc
Input file multipledataset2.nc has no raster bands.

and with the -debug mode

c:\>gdalwarp --debug on -of netCDF multipledataset1.nc dataset1.nc
GDAL_netCDF:
=====
calling nc_open( multipledataset1.nc )

GDAL_netCDF: driver detected file type=3, libnetcdf detected type=4
GDAL_netCDF: seting file type to 4, was 3
GDAL_netCDF: dim_count = 2
Warning 1: No UNIDATA NC_GLOBAL:Conventions attribute
GDAL_netCDF: var_count = 4
GDAL: GDALOpen(multipledataset1.nc, this=00000000005B6660) succeeds as netCDF.
Input file multipledataset1.nc has no raster bands.
GDAL: force close of multipledataset1.nc in GDALDriverManager cleanup.
GDAL: GDALClose(multipledataset1.nc, this=00000000005B6660)

comment:7 Changed 7 years ago by jluis

Description: modified (diff)

comment:8 Changed 7 years ago by etourigny

This confirms that Windows shared library handling does not have the same problem as in linux. Thanks Joaquim!

comment:9 Changed 7 years ago by etourigny

Cc: Even Rouault warmerdam added

Attaching a patch which resolves the issue in gdalwarp: exit(n) is replaced by GDALExit(n), which closes GDAL (and possibly OGR) ressources.

static int GDALExit( int nCode )
{
  const char  *pszDebug = CPLGetConfigOption("CPL_DEBUG",NULL);
  if( pszDebug && (EQUAL(pszDebug,"ON") || EQUAL(pszDebug,"") ) )
  {  
    GDALDumpOpenDatasets( stderr );
    CPLDumpSharedList( NULL );
  }

  GDALDestroyDriverManager();

#ifdef OGR_ENABLED
  OGRCleanupAll();
#endif

  exit( nCode );
}

In my opinion this should be added to the c api (in trunk and possibly other branches), and should be advertised in the gdal docs. This is an easy way to make sure datasets are closed automatically on exit.

Perhaps there are other gdal apps which could benefit from this.

Not commiting yet, waiting on input from others.

comment:10 Changed 7 years ago by etourigny

Component: defaultUtilities
Description: modified (diff)
Resolution: fixed
Status: newclosed
Summary: HDF5-DIAG Error detected when processing netcdf file with multiple datasetsgdalwarp does not properly close datasets in error conditions, can output errors in shared library unloading

Fixed in trunk (r24213) and 1.9(r24214).

Note: See TracTickets for help on using tickets.