Opened 12 years ago

Closed 10 years ago

#2270 closed enhancement (fixed)

Patch for HDF5 block size recognition

Reported by: jurien Owned by: chaitanya
Priority: normal Milestone: 1.6.3
Component: GDAL_Raster Version:
Severity: normal Keywords: HDF5
Cc: warmerdam, dnadeau, antonio

Description

Problem: HDF5 driver does not implement HDF chunk size detection. Simptoms: HDF5 is *very* slow on compressed files with chunksize != 1 line (default block size for HDF5 gdal driver)

I have modified the GDAL HDF5 driver to extract the chunk size and load it as the block size for the dataset.

Regards

Attachments (2)

gdal-1.5.0-HDF5-chunksize.patch (2.7 KB) - added by jurien 12 years ago.
patch to read chunksize from an HDF5 file and optimize reads
gdal-r17717-HDF5-chunksize.patch (3.1 KB) - added by antonio 10 years ago.
updated patch against r17717

Download all attachments as: .zip

Change History (15)

Changed 12 years ago by jurien

patch to read chunksize from an HDF5 file and optimize reads

comment:1 Changed 12 years ago by warmerdam

Cc: warmerdam dnadeau added
Component: defaultGDAL_Raster
Keywords: HDF5 added
Milestone: 1.5.2

Mateusz,

As time permits please review and apply the patches. I am not aware autotests for HDF5. This should also be addressed as part of this effort.

comment:2 Changed 12 years ago by warmerdam

Owner: changed from warmerdam to Mateusz Łoskot

comment:3 Changed 12 years ago by nowakpl

To me it looks like the patch would lead to incorrect memory layout for border blocks.

comment:4 Changed 12 years ago by jurien

@nowakpl: you mean if the blocksize is not a multiple of the imagesize? it is checked with at lines 304 and 308. I check for the size of the image and the boundaries of the block and keep the minimum of those. If that's not what you mean, please explain me what you meant so I can try and fix it.

Regards.

comment:5 Changed 11 years ago by Mateusz Łoskot

Owner: Mateusz Łoskot deleted

comment:6 Changed 11 years ago by jurien

Hello,

1.5.3 is about to be released, has this patch been postponed?

Regards

comment:7 Changed 11 years ago by warmerdam

Owner: set to warmerdam

Jurien,

I'm afraid this issue has fallen through the cracks. I'm not inclined to hold 1.5.3 back for this at this stage.

comment:8 Changed 11 years ago by antonio

Cc: antonio added

I'm massively using a modified version of the HDF5 driver (build as plugin) that includes the patch from jurien.

If this helps I can confirm that the patch works very well and also produces a good performances boost. Thanks jurien.

comment:9 Changed 10 years ago by jurien

Hello,

I'm bumping this ticket. I would really appreciate if the patch gets on the trunk.

To antonio: would you care to share your modifications on the HDF5 patch?

Regards.

comment:10 Changed 10 years ago by warmerdam

Milestone: 1.5.41.6.3
Owner: changed from warmerdam to chaitanya

Jurien,

Could you provide a pointer to a file demonstrating the issue?

Chaitanya,

Could you review the patch, and apply if it works ok? Please try to add/extend autotest if there is a file utilizing this feature that is a reasonable size.

Changed 10 years ago by antonio

updated patch against r17717

comment:11 in reply to:  9 Changed 10 years ago by antonio

Replying to jurien:

Hello,

I'm bumping this ticket. I would really appreciate if the patch gets on the trunk.

To antonio: would you care to share your modifications on the HDF5 patch?

Regards.

Hi jurien, I've just uploaded an updated version of your patch. It should fix the issue at image borders and it should also work a little better with non-chunked datasets. It is just an extract of my working code so please check it before applying.

comment:12 Changed 10 years ago by chaitanya

Status: newassigned

Tested and applied the patch in trunk (r17802) and 1.6 branch (r17801).

comment:13 Changed 10 years ago by Even Rouault

Resolution: fixed
Status: assignedclosed
Note: See TracTickets for help on using tickets.