Opened 15 years ago

Closed 13 years ago

#2270 closed enhancement (fixed)

Patch for HDF5 block size recognition

Reported by: jurien Owned by: chaitanya
Priority: normal Milestone: 1.6.3
Component: GDAL_Raster Version:
Severity: normal Keywords: HDF5
Cc: warmerdam, dnadeau, antonio

Description

Problem: HDF5 driver does not implement HDF chunk size detection. Simptoms: HDF5 is *very* slow on compressed files with chunksize != 1 line (default block size for HDF5 gdal driver)

I have modified the GDAL HDF5 driver to extract the chunk size and load it as the block size for the dataset.

Regards

Attachments (2)

gdal-1.5.0-HDF5-chunksize.patch (2.7 KB ) - added by jurien 15 years ago.
patch to read chunksize from an HDF5 file and optimize reads
gdal-r17717-HDF5-chunksize.patch (3.1 KB ) - added by antonio 13 years ago.
updated patch against r17717

Download all attachments as: .zip

Change History (15)

by jurien, 15 years ago

patch to read chunksize from an HDF5 file and optimize reads

comment:1 by warmerdam, 15 years ago

Cc: warmerdam dnadeau added
Component: defaultGDAL_Raster
Keywords: HDF5 added
Milestone: 1.5.2

Mateusz,

As time permits please review and apply the patches. I am not aware autotests for HDF5. This should also be addressed as part of this effort.

comment:2 by warmerdam, 15 years ago

Owner: changed from warmerdam to Mateusz Łoskot

comment:3 by nowakpl, 15 years ago

To me it looks like the patch would lead to incorrect memory layout for border blocks.

comment:4 by jurien, 15 years ago

@nowakpl: you mean if the blocksize is not a multiple of the imagesize? it is checked with at lines 304 and 308. I check for the size of the image and the boundaries of the block and keep the minimum of those. If that's not what you mean, please explain me what you meant so I can try and fix it.

Regards.

comment:5 by Mateusz Łoskot, 14 years ago

Owner: Mateusz Łoskot removed

comment:6 by jurien, 14 years ago

Hello,

1.5.3 is about to be released, has this patch been postponed?

Regards

comment:7 by warmerdam, 14 years ago

Owner: set to warmerdam

Jurien,

I'm afraid this issue has fallen through the cracks. I'm not inclined to hold 1.5.3 back for this at this stage.

comment:8 by antonio, 14 years ago

Cc: antonio added

I'm massively using a modified version of the HDF5 driver (build as plugin) that includes the patch from jurien.

If this helps I can confirm that the patch works very well and also produces a good performances boost. Thanks jurien.

comment:9 by jurien, 13 years ago

Hello,

I'm bumping this ticket. I would really appreciate if the patch gets on the trunk.

To antonio: would you care to share your modifications on the HDF5 patch?

Regards.

comment:10 by warmerdam, 13 years ago

Milestone: 1.5.41.6.3
Owner: changed from warmerdam to chaitanya

Jurien,

Could you provide a pointer to a file demonstrating the issue?

Chaitanya,

Could you review the patch, and apply if it works ok? Please try to add/extend autotest if there is a file utilizing this feature that is a reasonable size.

by antonio, 13 years ago

updated patch against r17717

in reply to:  9 comment:11 by antonio, 13 years ago

Replying to jurien:

Hello,

I'm bumping this ticket. I would really appreciate if the patch gets on the trunk.

To antonio: would you care to share your modifications on the HDF5 patch?

Regards.

Hi jurien, I've just uploaded an updated version of your patch. It should fix the issue at image borders and it should also work a little better with non-chunked datasets. It is just an extract of my working code so please check it before applying.

comment:12 by chaitanya, 13 years ago

Status: newassigned

Tested and applied the patch in trunk (r17802) and 1.6 branch (r17801).

comment:13 by Even Rouault, 13 years ago

Resolution: fixed
Status: assignedclosed
Note: See TracTickets for help on using tickets.