lack of closing read-only dataset dataset or calling GDALDestroyDriverManager() on NC4C (netCDF-4 classic model) file producing HDF5-DIAG error messages

[hsumanto]

Just trying to gdal.Open NC4C (netCDF-4 classic model) file and keep getting the below:

HDF5-DIAG: Error detected in HDF5 (1.8.8) thread 0:
  #000: H5T.c line 1721 in H5Tclose(): not a datatype
    major: Invalid arguments to routine
    minor: Inappropriate type

I attached the sample program getinfo and the sample dataset.

gdal.Open with ERROR

ncdump -k tile-nc4c.nc 
netCDF-4 classic model

getinfo tile-nc4c.nc 
Driver: netCDF/Network Common Data Format
Size is 2000x2000x1
HDF5-DIAG: Error detected in HDF5 (1.8.8) thread 0:
  #000: H5T.c line 1721 in H5Tclose(): not a datatype
    major: Invalid arguments to routine
    minor: Inappropriate type

gdal.Open OK

ncdump -k tile-nc.nc 
classic

getinfo tile-nc.nc 
Driver: netCDF/Network Common Data Format
Size is 333x333x1

[hsumanto]

getinfo program

[hsumanto]

sample dataset

[hsumanto]

I could reproduce this issue in HDF5 (1.8.6) as well.

[hsumanto]

Using gdal.Open in java

[hsumanto]

Using gdal.Open in java (See GdalTest.zip for example) to open NC4C file also produces the same error.

[Even Rouault]

Intuitively, I don't believe it is related to the Java bindings (I guess you get the same issue with gdalinfo ?). Not sure if it is in the netCDF driver or HDF5 driver either. CC'ing Etienne in case he's interested in that.

[etourigny]

gdalinfo can open both files fine, so the problem is probably with the java bindings.

Perhaps the hdf5 driver is being used instead of the netcdf driver? Try debug output in your test program and see if it really is the netcdf driver that opens the file (there is a lot of debug output from the netcdf driver).

Is there a java port of gdalinfo to test this file with?

Sorry I don't have any idea how to work with the java bindings nor the time to learn.

[hsumanto]

gdalinfo can open both files fine, so the problem is probably with the java bindings.

Like Even, I don't believe it is related to the java bindings too as the getinfo program (a simple program written in C++ which just uses gdal.Open) can produce the same message below:

HDF5-DIAG: Error detected in HDF5 (1.8.8) thread 0:
  #000: H5T.c line 1721 in H5Tclose(): not a datatype
    major: Invalid arguments to routine
    minor: Inappropriate type

Perhaps the hdf5 driver is being used instead of the netcdf driver? Try debug output in your test program and see if it really is the netcdf driver that opens the file (there is a lot of debug output from the netcdf driver).

The statement below:
printf( "Driver: %s/%s\n", GDALGetDriverShortName( hDriver ), GDALGetDriverLongName( hDriver ) );
printed out Driver: netCDF/Network Common Data Format, so it must be using netcdf driver.

The is the full content of the getinfo program:

#include <iostream>
#include "gdal_priv.h"
#include "exceptions.h"

namespace VPAC {

/**
 * Get info from raster.
 */
class GetInfoTest {
protected:
	GDALDataset *input;

public:
	std::string inputFileName;
	std::string outputFileName;

public:

	GetInfoTest() :
	input(NULL)
	{
	}

	~GetInfoTest() throw() {
		if (input != NULL) {
			//GDALClose(input);
			//input = NULL;
		}
	}

	void run() {
		openDatasets();
	}

protected:

	void openDatasets() throw (MyException) {
		if (input != NULL) {
			throw MyException("Datasets are already open!");
		}

	    	input = (GDALDataset *) GDALOpen(inputFileName.c_str(), GA_ReadOnly);
	    	if (input == NULL) {
	    		throw RequestException("Could not open source file.");
	    	}
    	    	GDALDriverH   hDriver;

	    	hDriver = GDALGetDatasetDriver( input );
            	printf( "Driver: %s/%s\n",
            	GDALGetDriverShortName( hDriver ),
            	GDALGetDriverLongName( hDriver ) );

    	    	printf( "Size is %dx%dx%d\n",
            		GDALGetRasterXSize( input ), 
            		GDALGetRasterYSize( input ),
            		GDALGetRasterCount( input ) );
	}
}; // class GetInfoTest

} // namespace VPAC

using namespace VPAC;

int main(int argc, char *argv[]) {
	try {
		GetInfoTest getinfo;
		GDALAllRegister();

		if (argc != 2) {
			throw RequestException("Invalid arguments.");
		}

		getinfo.inputFileName = argv[1];
		getinfo.run();

	} catch (RequestException e) {
		std::cerr << "Error: " << e.what() << std::endl;
		std::cerr << "Usage: getinfo <input file>" <<
				std::endl;
	} catch (MyException e) {
		std::cerr << "Error: " << e.what() << std::endl;
	}
    return 0;
}

[etourigny]

Please test against gdalinfo (java version), as gdalinfo (c version) does not report the error.

The errors you report are with programs written by your organisation, not by the official gdalinfo in c or java.

The lack of GDALClose( ) in gdalinfo.cpp causes the error, if you uncomment the line GDALClose(input); the error does not appear. I suspect it is the lack hDataset.delete() in GDALTest.java that is causing the error.

[hsumanto]

Yeah I noticed that during testing as well. Having GDALClose( ) in gdalinfo.cpp and having hDataset.delete() in GDALTest.java fix the error messages which appear when opening NC4C (netCDF-4 classic model).

The only thing which still left me puzzled at that time:
When opening NC file (tile-nc.nc in sample_dataset.zip), not having GDALClose( ) in gdalinfo.cpp and not having hDataset.delete() in GDALTest.java -> No error messages are encountered.
When opening NC4C file (tile-nc4c.nc in sample_dataset.zip), not having GDALClose( ) in gdalinfo.cpp and not having hDataset.delete() in GDALTest.java -> HDF5-DIAG error messages are shown.

In both cases (gdalinfo.cpp and GDALTest.java), gdal is opening the dataset in readonly mode (not update mode), thus I was assuming that we won't have to explicitly close a dataset opened in readonly mode. Therefore, I was just wondering whether I should report this inconsistency as a ticket.

Thanks for the quick reply Etienne, really appreciate it :)

public static Dataset Open(String name, int eAccess)

    Open a raster file as a Dataset object.

    This function will try to open the passed file, or virtual dataset name by invoking the Open method of each registered Driver in turn. The first successful open will result in a returned dataset. If all drivers fail then null is returned.

    It is required that you explicitly close a dataset opened in update mode with the Dataset.delete() method. Otherwise the data might not be flushed to the disk. Don't rely only on Java garbage collection.

    Parameters:
        name - the name of the file to access. In the case of exotic drivers this may not refer to a physical file, but instead contain information for the driver on how to access a dataset.
        eAccess - the desired access, either gdalconst.GA_Update or gdalconst.GA_ReadOnly. Many drivers support only read only access. 
    Returns:
        A Dataset object or null on failure.

[etourigny]

Further investigation reveals that the lack of calling GDALDestroyDriverManager() and GDALClose( hDataset ) cause the hdf5 errors. By using one or the other the error does not appear.

The same is probably true for your java program, which is lacking gdal.GDALDestroyDriverManager() as well hDataset.delete();

I don't see how to fix the netcdf driver for this case as the internals of netcdf-4/hdf5 are rather complex.

Closing this bug as invalid, as the recommendation is to close the dataset after usage, or using GDALDestroyDriverManager() - which in 1.9 properly closes all open datasets.