Opened 12 years ago
Closed 12 years ago
#4554 closed defect (fixed)
gdalwarp does not properly close datasets in error conditions, can output errors in shared library unloading — at Version 10
Reported by: | hsumanto | Owned by: | warmerdam |
---|---|---|---|
Priority: | normal | Milestone: | |
Component: | Utilities | Version: | unspecified |
Severity: | normal | Keywords: | netcdf |
Cc: | etourigny, jluis, Even Rouault, warmerdam |
Description (last modified by )
I have two netcdf files containing multiple datasets. The netCDF driver produces some HDF5-DIAG error message on the first file but on the second file, the driver handles it without any error.
gdalwarp -of netCDF multipledataset1.nc dataset1.nc Warning 1: No UNIDATA NC_GLOBAL:Conventions attribute Input file multipledataset1.nc has no raster bands. HDF5-DIAG: Error detected in HDF5 (1.8.8) thread 0: #000: H5T.c line 1721 in H5Tclose(): not a datatype major: Invalid arguments to routine minor: Inappropriate type gdalwarp -of netCDF multipledataset2.nc dataset2.nc2 Input file multipledataset2.nc has no raster bands.
Previous summary was "HDF5-DIAG Error detected when processing netcdf file with multiple datasets"
Change History (11)
by , 12 years ago
Attachment: | dataset.zip added |
---|
comment:1 by , 12 years ago
Cc: | added |
---|
comment:2 by , 12 years ago
GDAL treats both files as having subdatasets (with no bands), therefore gdalwarp cannot operate on them, you have to work with the subdatasets.
The error for the first file, as well as the debugging output (GDAL: force close of multipledataset2.nc in GDALDriverManager cleanup.) is cause for minor concern, I will see what is wrong. It seems the input file is not closed properly, and is unrelated to the lack of support for gdalwarp with multiple subdatasets.
One way or another, don't count on having the fundamental problem fixed soon though (see my email to the list today in answer to Hendy's post).
$ gdalwarp --debug on -of netCDF multipledataset1.nc dataset1.nc GDAL_netCDF: ===== calling nc_open( multipledataset1.nc ) GDAL_netCDF: driver detected file type=3, libnetcdf detected type=4 GDAL_netCDF: seting file type to 4, was 3 GDAL_netCDF: dim_count = 2 Warning 1: No UNIDATA NC_GLOBAL:Conventions attribute GDAL_netCDF: var_count = 4 GDAL: GDALOpen(multipledataset1.nc, this=0x17656a0) succeeds as netCDF. Input file multipledataset1.nc has no raster bands. GDAL: In GDALDestroy - unloading GDAL shared library. GDAL: force close of multipledataset1.nc in GDALDriverManager cleanup. HDF5-DIAG: Error detected in HDF5 (1.8.8) thread 0: #000: H5T.c line 1721 in H5Tclose(): not a datatype major: Invalid arguments to routine minor: Inappropriate type GDAL: GDALClose(multipledataset1.nc, this=0x17656a0) $ gdalwarp --debug on -of netCDF multipledataset2.nc dataset2.nc2 GDAL_netCDF: ===== calling nc_open( multipledataset2.nc ) GDAL_netCDF: driver detected file type=1, libnetcdf detected type=1 GDAL_netCDF: dim_count = 2 GDAL_netCDF: var_count = 9 GDAL_netCDF: variable #7 [lat] was ignored GDAL_netCDF: variable #8 [lon] was ignored GDAL: GDALOpen(multipledataset2.nc, this=0x12666a0) succeeds as netCDF. Input file multipledataset2.nc has no raster bands. GDAL: In GDALDestroy - unloading GDAL shared library. GDAL: force close of multipledataset2.nc in GDALDriverManager cleanup. GDAL: GDALClose(multipledataset2.nc, this=0x12666a0)
follow-up: 5 comment:4 by , 12 years ago
Cc: | added |
---|
With much help from Even and Frank, the source of the error message has been found.
It seems that the hdf5 library uses atextit() exit handlers (specifically the function H5_term_library()) which get called before gdal unloads. The call to nc_close (ehich calls H5close()) raises an error because the hdf5 library has been unloaded.
Disabling most of gdaldllmain.cpp seems to solve it temporarily, but a better solution should be found.
//#ifdef __GNUC__ #ifdef notdef
Not sure if this happens in Windows - adding Joaquim in cc... Can you see if you get the same error?
comment:5 by , 12 years ago
Description: | modified (diff) |
---|
Replying to etourigny:
With much help from Even and Frank, the source of the error message has been found.
It seems that the hdf5 library uses atextit() exit handlers (specifically the function H5_term_library()) which get called before gdal unloads. The call to nc_close (ehich calls H5close()) raises an error because the hdf5 library has been unloaded.
Disabling most of gdaldllmain.cpp seems to solve it temporarily, but a better solution should be found.
//#ifdef __GNUC__ #ifdef notdefNot sure if this happens in Windows - adding Joaquim in cc... Can you see if you get the same error?
comment:6 by , 12 years ago
Description: | modified (diff) |
---|
Etienne,
On Win7 64 with HDF5 1.8.8 and netCDF 4.1.3 I get this
c:\>gdalwarp -of netCDF multipledataset1.nc dataset1.nc Warning 1: No UNIDATA NC_GLOBAL:Conventions attribute Input file multipledataset1.nc has no raster bands. c:\>gdalwarp -of netCDF multipledataset2.nc dataset1.nc Input file multipledataset2.nc has no raster bands.
and with the -debug mode
c:\>gdalwarp --debug on -of netCDF multipledataset1.nc dataset1.nc GDAL_netCDF: ===== calling nc_open( multipledataset1.nc ) GDAL_netCDF: driver detected file type=3, libnetcdf detected type=4 GDAL_netCDF: seting file type to 4, was 3 GDAL_netCDF: dim_count = 2 Warning 1: No UNIDATA NC_GLOBAL:Conventions attribute GDAL_netCDF: var_count = 4 GDAL: GDALOpen(multipledataset1.nc, this=00000000005B6660) succeeds as netCDF. Input file multipledataset1.nc has no raster bands. GDAL: force close of multipledataset1.nc in GDALDriverManager cleanup. GDAL: GDALClose(multipledataset1.nc, this=00000000005B6660)
comment:7 by , 12 years ago
Description: | modified (diff) |
---|
comment:8 by , 12 years ago
This confirms that Windows shared library handling does not have the same problem as in linux. Thanks Joaquim!
comment:9 by , 12 years ago
Cc: | added |
---|
Attaching a patch which resolves the issue in gdalwarp: exit(n) is replaced by GDALExit(n), which closes GDAL (and possibly OGR) ressources.
static int GDALExit( int nCode ) { const char *pszDebug = CPLGetConfigOption("CPL_DEBUG",NULL); if( pszDebug && (EQUAL(pszDebug,"ON") || EQUAL(pszDebug,"") ) ) { GDALDumpOpenDatasets( stderr ); CPLDumpSharedList( NULL ); } GDALDestroyDriverManager(); #ifdef OGR_ENABLED OGRCleanupAll(); #endif exit( nCode ); }
In my opinion this should be added to the c api (in trunk and possibly other branches), and should be advertised in the gdal docs. This is an easy way to make sure datasets are closed automatically on exit.
Perhaps there are other gdal apps which could benefit from this.
Not commiting yet, waiting on input from others.
comment:10 by , 12 years ago
Component: | default → Utilities |
---|---|
Description: | modified (diff) |
Resolution: | → fixed |
Status: | new → closed |
Summary: | HDF5-DIAG Error detected when processing netcdf file with multiple datasets → gdalwarp does not properly close datasets in error conditions, can output errors in shared library unloading |
sample dataset used