Opened 11 years ago

Closed 11 years ago

#2591 closed defect (fixed)

NetCDF Driver should read/write data in stripes instead of the whole array

Reported by: rsignell Owned by: warmerdam
Priority: normal Milestone: 1.5.3
Component: GDAL_Raster Version: 1.5.0
Severity: normal Keywords: netcdf
Cc: dnadeau

Description (last modified by warmerdam)

When I tried to convert a 20000x15000 file using the GMT driver (which writes NetCDF with Z as a 1D variable) it works fine, but the NetCDF driver gives an "Out of memory" error:

gdal_translate psdem_2005_geo.tif -of netcdf psdem_2005.nc
Input file size is 20000, 15000
0ERROR 2: CPLMalloc(): Out of memory allocating 1200000000 bytes.

It looks like the NetCDF driver is trying to allocate memory for the entire data grid being processed instead of reading/writing by column, row or chunk. This should be fixed, to allow processing of much larger grids on machines with limited memory.

Change History (2)

comment:1 Changed 11 years ago by warmerdam

Cc: dnadeau added
Component: defaultGDAL_Raster
Description: modified (diff)
Keywords: netcdf added
Priority: highnormal
Status: newassigned

Denis,

Any thoughts on this issue? Skimming the CreateCopy? code in the netcdf driver, it looks like things are done one scanline at a time which should be OK.

I'll try and reproduce ...

comment:2 Changed 11 years ago by warmerdam

Milestone: 1.5.3
Resolution: fixed
Status: assignedclosed
Version: unspecified1.5.0

It turns out the data was being written one line at a time, but for no good apparent reason the working buffer was also dimensioned by the number of lines. I have corrected this in trunk (r15476) and 1.5 branch (r15477).

Note: See TracTickets for help on using tickets.