Opened 10 years ago

Closed 10 years ago

#4404 closed defect (fixed)

gdal_translate GeoTIFF to NetCDF with NC4 DEFLATE compression is taking forever

Reported by: hsumanto Owned by: warmerdam
Priority: normal Milestone:
Component: default Version: unspecified
Severity: normal Keywords: netcdf
Cc: etourigny

Description

I performing translation from geotiff to netcdf with NC4 DEFLATE compression option and it is taking so long, not sure what is going wrong here. My machine has HDF5 (1.8.6) installed.

gdal_translate -of netCDF -co COMPRESS=DEFLATE -co ZLEVEL=6 -co FORMAT=NC4C -co WRITE_GDAL_TAGS=yes tile_x5_y2.tif compressed_tile.nc

Sometimes the below message was printed. HDF5-DIAG: Error detected in HDF5 (1.8.6)

Attachments (1)

tile_x5_y2.tif (588.7 KB) - added by hsumanto 10 years ago.
geotiff to be converted to netcdf

Download all attachments as: .zip

Change History (6)

Changed 10 years ago by hsumanto

Attachment: tile_x5_y2.tif added

geotiff to be converted to netcdf

comment:1 Changed 10 years ago by hsumanto

Managed to reproduce the same issue on a different machine with HDF5 (1.8.4-patch1) installed.

HDF5-DIAG: Error detected in HDF5 (1.8.4-patch1) thread 0:

#000: ../../../src/H5T.c line 1705 in H5Tclose(): not a datatype

major: Invalid arguments to routine minor: Inappropriate type

HDF5-DIAG: Error detected in HDF5 (1.8.4-patch1) thread 0:

#000: ../../../src/H5T.c line 1705 in H5Tclose(): not a datatype

major: Invalid arguments to routine minor: Inappropriate type

HDF5-DIAG: Error detected in HDF5 (1.8.4-patch1) thread 0:

#000: ../../../src/H5T.c line 1705 in H5Tclose(): not a datatype

major: Invalid arguments to routine minor: Inappropriate type

HDF5-DIAG: Error detected in HDF5 (1.8.4-patch1) thread 0:

#000: ../../../src/H5T.c line 1705 in H5Tclose(): not a datatype

major: Invalid arguments to routine minor: Inappropriate type

HDF5-DIAG: Error detected in HDF5 (1.8.4-patch1) thread 0:

#000: ../../../src/H5T.c line 1705 in H5Tclose(): not a datatype

major: Invalid arguments to routine minor: Inappropriate type

HDF5-DIAG: Error detected in HDF5 (1.8.4-patch1) thread 0:

#000: ../../../src/H5T.c line 1705 in H5Tclose(): not a datatype

major: Invalid arguments to routine minor: Inappropriate type

comment:2 Changed 10 years ago by Even Rouault

Cc: etourigny added

comment:3 Changed 10 years ago by etourigny

This stupid error *should* have been detected by netcdf_21 test, but it was disabled in my system because it takes a long time and is skipped unless GDAL_RUN_SLOW_TESTS=YES .

It slipped in there somehow during the refactor (r23456), fix is quite simple.

Fixed in r23635

This also seems to fix the problem with gdalwarp related described in #4221 (comment 7). Bonus side effect: it seems that using gdalwarp with DEFLATE compression using NC4C format is much more size-efficient (half the size in 1 test) than with the gtiff driver, and cpu time is only slightly slower (58 vs. 51 s).

Hendy - please test this works for you and also trying large gdalwarp operations with the netcdf driver and COMPRESS=DEFLATE (with zlevel=6 and zlevel=9), thanks.

comment:4 Changed 10 years ago by hsumanto

I was wondering about it as well as we have fixed this issue last time in git repository but why it reappears again.

Thanks Etienne for adding the fix back to trunk.

I have tested the gdalwarp with DEFLATE compression using NC4C format for LANDSAT dataset, it seems to be working without any issue now.

comment:5 Changed 10 years ago by etourigny

Resolution: fixed
Status: newclosed

The error slipped in unnoticed, thanks for your help.

Note: See TracTickets for help on using tickets.