gdal_translate GeoTIFF to NetCDF with NC4 DEFLATE compression is taking forever

[hsumanto]

I performing translation from geotiff to netcdf with NC4 DEFLATE compression option and it is taking so long, not sure what is going wrong here. My machine has HDF5 (1.8.6) installed.

gdal_translate -of netCDF -co COMPRESS=DEFLATE -co ZLEVEL=6 -co FORMAT=NC4C -co WRITE_GDAL_TAGS=yes tile_x5_y2.tif compressed_tile.nc

Sometimes the below message was printed. HDF5-DIAG: Error detected in HDF5 (1.8.6)

[hsumanto]

geotiff to be converted to netcdf

[hsumanto]

Managed to reproduce the same issue on a different machine with HDF5 (1.8.4-patch1) installed.

HDF5-DIAG: Error detected in HDF5 (1.8.4-patch1) thread 0:

#000: ../../../src/H5T.c line 1705 in H5Tclose(): not a datatype

major: Invalid arguments to routine minor: Inappropriate type

HDF5-DIAG: Error detected in HDF5 (1.8.4-patch1) thread 0:

#000: ../../../src/H5T.c line 1705 in H5Tclose(): not a datatype

major: Invalid arguments to routine minor: Inappropriate type

HDF5-DIAG: Error detected in HDF5 (1.8.4-patch1) thread 0:

#000: ../../../src/H5T.c line 1705 in H5Tclose(): not a datatype

major: Invalid arguments to routine minor: Inappropriate type

HDF5-DIAG: Error detected in HDF5 (1.8.4-patch1) thread 0:

#000: ../../../src/H5T.c line 1705 in H5Tclose(): not a datatype

major: Invalid arguments to routine minor: Inappropriate type

HDF5-DIAG: Error detected in HDF5 (1.8.4-patch1) thread 0:

#000: ../../../src/H5T.c line 1705 in H5Tclose(): not a datatype

major: Invalid arguments to routine minor: Inappropriate type

HDF5-DIAG: Error detected in HDF5 (1.8.4-patch1) thread 0:

#000: ../../../src/H5T.c line 1705 in H5Tclose(): not a datatype

major: Invalid arguments to routine minor: Inappropriate type

[etourigny]

This stupid error *should* have been detected by netcdf_21 test, but it was disabled in my system because it takes a long time and is skipped unless GDAL_RUN_SLOW_TESTS=YES .

It slipped in there somehow during the refactor (r23456), fix is quite simple.

Fixed in r23635

This also seems to fix the problem with gdalwarp related described in #4221 (comment 7). Bonus side effect: it seems that using gdalwarp with DEFLATE compression using NC4C format is much more size-efficient (half the size in 1 test) than with the gtiff driver, and cpu time is only slightly slower (58 vs. 51 s).

Hendy - please test this works for you and also trying large gdalwarp operations with the netcdf driver and COMPRESS=DEFLATE (with zlevel=6 and zlevel=9), thanks.

[hsumanto]

I was wondering about it as well as we have fixed this issue last time in git repository but why it reappears again.

Thanks Etienne for adding the fix back to trunk.

I have tested the gdalwarp with DEFLATE compression using NC4C format for LANDSAT dataset, it seems to be working without any issue now.

[etourigny]

The error slipped in unnoticed, thanks for your help.