Opened 11 years ago

Closed 10 years ago

#3976 closed defect (fixed)

Problem creating netcdf files with numerous bands

Reported by: tbouffon Owned by: Kyle Shannon
Priority: normal Milestone: 1.9.0
Component: GDAL_Raster Version: 1.8.0
Severity: normal Keywords: raster, netcdf, gdal_translate
Cc: Kyle Shannon

Description

gdal_translate has trouble creating netcdf files with more than 2053 bands :

$ gdal_translate -of NETCDF 2053.grb 2053.nc
Input file size is 1, 1
$ gdal_translate -of NETCDF 2054.grb 2054.nc 2>&1 | head
Input file size is 1, 1
0ERROR 5: Access window out of range in RasterIO().  Requested
(0,1) of size 1x1 on raster of 1x1.
ERROR 5: Access window out of range in RasterIO().  Requested
(0,1) of size 1x1 on raster of 1x1.
ERROR 5: Access window out of range in RasterIO().  Requested
(0,1) of size 1x1 on raster of 1x1.
ERROR 5: Access window out of range in RasterIO().  Requested
(0,1) of size 1x1 on raster of 1x1.
ERROR 5: Access window out of range in RasterIO().  Requested

Conversion to Geotiff from the same files do not present any problem. Is there a hard limit for netcdf creation ?

Attachments (2)

2053.grb (220.5 KB) - added by tbouffon 11 years ago.
grib file with 2053 bands
2054.grb (220.6 KB) - added by tbouffon 11 years ago.
Grib file with 2054 bands which causes troubles

Download all attachments as: .zip

Change History (10)

Changed 11 years ago by tbouffon

Attachment: 2053.grb added

grib file with 2053 bands

Changed 11 years ago by tbouffon

Attachment: 2054.grb added

Grib file with 2054 bands which causes troubles

comment:1 Changed 11 years ago by Even Rouault

Cc: Kyle Shannon added

comment:2 Changed 11 years ago by Kyle Shannon

Owner: changed from warmerdam to Kyle Shannon
Status: newassigned

What are the particulars of your netcdf build? Version, platform, etc. If possible, run nc-config --all. I didn't have a problem creating the nc file on my build. I will look into this further.

kss

comment:3 Changed 11 years ago by tbouffon

I don't have nc-config. My netcdf is 3.6.2 built on x86_64 rhel 4.8 netcdf has been built with :

./configure --enable-shared

I managed to translate the file on a fedora box, too. I'm trying to build gdal 1.8 against a different netcdf, to see if it changes something.

Thanks, Thomas

comment:4 Changed 11 years ago by tbouffon

Version: 1.7.11.8.0

HI, I rebuilt gdal 1.8 statically linked against netcdf 4.1.1. I stillo have the same error. nc-config from this netcdf install gives

  --cc        -> cc
  --cflags    ->  -I/home/mfdp/mfdp013/netcdf/include
  --libs      -> -L/home/mfdp/mfdp013/netcdf/lib -lnetcdf -L/usr/lib64 -lcurl -L/usr/kerberos/lib -L/usr/kerberos/lib64 -lssl -lcrypto -lgssapi_krb5 -lkrb5 -lcom_err -lk5crypto -lresolv -ldl -lz -lgssapi_krb5 -lkrb5 -lk5crypto -lcom_err -lresolv -L/usr/kerberos/lib -L/usr/kerberos/lib64 -lidn -lssl -lcrypto -lssl -lcrypto -lgssapi_krb5 -lkrb5 -lcom_err -lk5crypto -lresolv -ldl -lz -lz

  --cxx       -> c++
  --has-c++   -> yes

  --fc        -> gfortran
  --fflags    -> -g -O2 -I/home/mfdp/mfdp013/netcdf/include
  --flibs     -> -L/home/mfdp/mfdp013/netcdf/lib -lnetcdf -L/usr/lib64 -lcurl -L/usr/kerberos/lib -L/usr/kerberos/lib64 -lssl -lcrypto -lgssapi_krb5 -lkrb5 -lcom_err -lk5crypto -lresolv -ldl -lz -lgssapi_krb5 -lkrb5 -lk5crypto -lcom_err -lresolv -L/usr/kerberos/lib -L/usr/kerberos/lib64 -lidn -lssl -lcrypto -lssl -lcrypto -lgssapi_krb5 -lkrb5 -lcom_err -lk5crypto -lresolv -ldl -lz -lz
  --has-f77   -> yes
  --has-f90   -> yes

  --has-dap   -> yes
  --has-nc2   -> yes
  --has-nc4   -> no
  --has-hdf5  -> no
  --has-hdf4  -> no
  --has-szlib -> no

  --prefix    -> /home/mfdp/mfdp013/netcdf
  --includedir-> /home/mfdp/mfdp013/netcdf/include
  --version   -> netCDF 4.1.1

I'm building gdal with rpmbuild, which adds lots of options to configure. I'm going to try and build it manually.

comment:5 Changed 11 years ago by tbouffon

I seem to have found the problem : on line 2355 in frmts/netcdf/netcdf.cpp, anBandMap is initialized as an array of NC_MAX_DIMS integers, and on line 2662 a loop through the bands writes anBandMap[j-1]=j; These are the only references to this variable in the whole gdal tarball.

So I removed the definition of anBandMap and the loop and everything worked now.

I guess the strange number of bands needed to have gdal_translate are due to the way gcc organizes compiled code : with different -O and -g combinations, the results where varying from segmentation faults to incomplete files with damaged geography.

Moreover, the bands are registered as variables, so the relevant constant should be NC_MAX_VARS, which is never used.

Is this reference to anBandMap a remnant from an old, unused part of code, or a typo against panBandMap ? Why is it never used ?

Cheers, Thomas

comment:6 Changed 11 years ago by Kyle Shannon

Thanks for investigating. I am not sure why that is the way it is. I agree with the logic above, referring to vars vs dims. I will check on it and update the ticket soon, I am a little under the weather, so it probably won't be until next week.

kss

comment:7 Changed 10 years ago by Kyle Shannon

Milestone: 1.9.0

After looking at this a bit, I am not certain what the anBandMap is for or why it was implemented. Bands may or may not represent variables, such as when the data dimensions are x, y, and time. The bands would represent a dimension. I am not sure if the band map is there for a reason or not. If I can't find a valid reason for having it, I will comment it out. Thanks again, and sorry for the delay.

kss

comment:8 Changed 10 years ago by etourigny

Resolution: fixed
Status: assignedclosed

I can't reproduce the error in gdal-1.8.1 (which has anBandMap).

It has been removed entirely in trunk (r23265).

Closing this bug, if the problems comes back please reopen.

Note: See TracTickets for help on using tickets.